A critical review of “Evaluation in Participatory Design : A Literature Survey”
Bossen, Dindler & Iversen (2016) consider how Participatory Design (PD) projects are evaluated. The authors introduce Evaluation Theory and highlight the central aims of PD as being mutual learning, empowerment, democracy and workplace quality. From a survey of core PD publications, the authors select papers that deal with “grand questions” in PD and that actively engaged with evaluation of PD. The authors identify 7 questions for assessing PD evaluations and ask these questions of their selected papers. The authors call for; 1) explicit and systematic evaluation (to enhance knowledge and learning and support funding applications); 2) clarity in methods and analysis (papers appear to ‘start from scratch’ rather than extending existing literature); 3) frameworks (theory exists, but is rarely applied); and 4) democratisation, empowerment and mutual learning within the evaluation process as well as the PD process itself (participants appear to be interviewed by researchers as a post-project administrative task for control and accountability rather than as a knowledge and learning tool).
The call for explicit and systematic evaluation that builds on previous work in the field is a sensible move towards developing best practices in PD. This could be a useful learning and development pathway for individual, organisational and community practices within PD.
The call for participatory evaluation to embrace the central aims of the PD concept itself is interesting. Perhaps there is a role for incorporating participatory evaluation within the PD process itself rather than simply as an administrative reporting tool. From this perspective, PD could be viewed as an iterative learning process based on the adult learning cycle (Kolb, 1984) rather than as a linear process to solve a design problem. Such an approach has been proposed for addressing wicked problems as an open, inclusive and iterative inquiry embracing imagination and critical rationality (Russell, 2010).
The authors acknowledge that the selected papers are targeted at the research community and that it is possible that evaluation and feedback are both implicitly and explicitly performed with participants through PD processes themselves and that other channels may be used for reporting back to stakeholders and decision-makers. Making this explicit in future studies and reporting evaluation from these different perspectives would allow richer conclusions to be drawn on outcomes, outputs and impacts of PD projects. Taking this further, an alternative to providing feedback along traditional reporting lines is to include stakeholders and decision-makers with the participants in collective design processes. This has been found to be effective in other fields (for a Marine Spatial Planning example, see Orchard-Webb et al., 2016).
The selection of papers to analyse is narrowly and inconsistently defined and the authors acknowledge that their method may exclude a significant part of the PD research community. Nevertheless, the authors make useful proposals for improving the evaluation of PD. I believe that this is an avenue worth exploring that has the potential to improve the PD process and its deliverables for all parties.
Bossen, C., Dindler, C., Iversen, O.S., 2016. Evaluation in Participatory Design : A Literature Survey. Particip. Des. Conf. 151–160. doi:10.1145/2940299.2940303
Kolb, D.A., 1984. Experiential learning: Experience as the source of learning and development, David A. Kolb, Prentice-Hall International, Hemel Hempstead, Herts., 1984. No. of pages: xiii + 256. J. Organ. Behav. 8, 359–360. doi:10.1002/job.4030080408
Orchard-Webb, J., Kenter, J.O., Bryce, R., Church, A., 2016. Deliberative Democratic Monetary Valuation to implement the Ecosystem Approach. Ecosyst. Serv. 21, 308–318. doi:https://doi.org/10.1016/j.ecoser.2016.09.005
Russell, J., 2010. A philosophical framework for an open and critical transdisciplinary inquiry, in: Brown, V., Harris, J., Russell, J. (Eds.), Tackling Wicked Problems: Through the Transdisciplinary Imagination. Earthscan, New York, p. 312.
Author: Matthew Snape