A critical review of “Evaluation in Participatory Design : A Literature Survey”

Bossen, Dindler & Iversen (2016) consider how Participatory Design (PD) projects are evaluated. The authors introduce Evaluation Theory and highlight the central aims of PD as being mutual learning, empowerment, democracy and workplace quality. From a survey of core PD publications, the authors select papers that deal with “grand questions” in PD and that actively engaged with evaluation of PD. The authors identify 7 questions for assessing PD evaluations and ask these questions of their selected papers. The authors call for; 1) explicit and systematic evaluation (to enhance knowledge and learning and support funding applications); 2) clarity in methods and analysis (papers appear to ‘start from scratch’ rather than extending existing literature); 3) frameworks (theory exists, but is rarely applied); and 4) democratisation, empowerment and mutual learning within the evaluation process as well as the PD process itself (participants appear to be interviewed by researchers as a post-project administrative task for control and accountability rather than as a knowledge and learning tool).

The call for explicit and systematic evaluation that builds on previous work in the field is a sensible move towards developing best practices in PD. This could be a useful learning and development pathway for individual, organisational and community practices within PD.

The call for participatory evaluation to embrace the central aims of the PD concept itself is interesting. Perhaps there is a role for incorporating participatory evaluation within the PD process itself rather than simply as an administrative reporting tool. From this perspective, PD could be viewed as an iterative learning process based on the adult learning cycle (Kolb, 1984) rather than as a linear process to solve a design problem. Such an approach has been proposed for addressing wicked problems as an open, inclusive and iterative inquiry embracing imagination and critical rationality (Russell, 2010).

The authors acknowledge that the selected papers are targeted at the research community and that it is possible that evaluation and feedback are both implicitly and explicitly performed with participants through PD processes themselves and that other channels may be used for reporting back to stakeholders and decision-makers. Making this explicit in future studies and reporting evaluation from these different perspectives would allow richer conclusions to be drawn on outcomes, outputs and impacts of PD projects. Taking this further, an alternative to providing feedback along traditional reporting lines is to include stakeholders and decision-makers with the participants in collective design processes. This has been found to be effective in other fields (for a Marine Spatial Planning example, see Orchard-Webb et al., 2016).

The selection of papers to analyse is narrowly and inconsistently defined and the authors acknowledge that their method may exclude a significant part of the PD research community. Nevertheless, the authors make useful proposals for improving the evaluation of PD. I believe that this is an avenue worth exploring that has the potential to improve the PD process and its deliverables for all parties.

 

References

Bossen, C., Dindler, C., Iversen, O.S., 2016. Evaluation in Participatory Design : A Literature Survey. Particip. Des. Conf. 151–160. doi:10.1145/2940299.2940303

Kolb, D.A., 1984. Experiential learning: Experience as the source of learning and development, David A. Kolb, Prentice-Hall International, Hemel Hempstead, Herts., 1984. No. of pages: xiii + 256. J. Organ. Behav. 8, 359–360. doi:10.1002/job.4030080408

Orchard-Webb, J., Kenter, J.O., Bryce, R., Church, A., 2016. Deliberative Democratic Monetary Valuation to implement the Ecosystem Approach. Ecosyst. Serv. 21, 308–318. doi:https://doi.org/10.1016/j.ecoser.2016.09.005

Russell, J., 2010. A philosophical framework for an open and critical transdisciplinary inquiry, in: Brown, V., Harris, J., Russell, J. (Eds.), Tackling Wicked Problems: Through the Transdisciplinary Imagination. Earthscan, New York, p. 312.


Author: Matthew Snape

One response to “A critical review of “Evaluation in Participatory Design : A Literature Survey””

  1. Simon Bowen says:

    Thanks, Matthew. A good summary of the paper. The link you make with Kolb’s adult learning cycles is interesting. I can also see parallel links with Action Research (cycles of plan-act-observe-reflect). Evaluation within the PD process, as opposed to at its end, also relates the Bossen et al.’s discussion of formative rather than summative evaluation, and their framing questions “who conducts the evaluation by the participation of whom for which purposes and uses and based on which criteria?”. Bossen et al. suggest that evaluation within a PD project will increase the effectiveness of that project (not simply measure this effectiveness). But, could there be limitations to this? In what ways can design participants undertake evaluation activities that visibly move the design project forward not just evaluate it? Could evaluation be a distraction from design? Can the two complement each other?

Leave a Reply

Your email address will not be published. Required fields are marked *