Effectiveness of systemic innovations
Evaluating Systemic Impact of Innovations
People involved
Hannah Goss (me)
Rick Schifferstein
Jotte de Koning
Nynke Tromp
FETE Consortium
Project Output
Journal Paper
When
During PhD
Challenge
Assessing impact within societal contexts remains a significant challenge in design. Interventions often result in impacts that are difficult to observe or measure, such as behavioural changes, improvements in well-being, or increased resilience. As such, designers aiming to foster systemic change face immense challenges in assessing the effectiveness of their interventions on actually achieving desired aims. This is particularly challenging in transitions where change unfolds over decades. Existing evaluation methods often fail to capture the complexity and multi-scalar dynamics inherent in systemic change processes.
This case addresses the pressing question:
What is the perceived effectiveness of transition design interventions resulting from a transition design process?
Once interventions are designed, efforts to assess their impact require criteria and approaches that reflect the dynamics and evolving nature of the context. As transition design is a young field, a common evaluation method has yet to emerge. This case lays the foundation for transition design evaluation.
Approach
Mixed Methods
Narrative-based
Survey with 312 people
Statistics
To tackle the challenge I developed a Transition Design Activities and Outcomes Framework that outlines five essential transition design activities to three evaluative criteria for the outcomes.
Transition Design Activities:
Navigating scales from micro to macro levels
Considering temporality from present to far future
Engaging and repositioning actors from individual actors to actor networks
Framing and designing from single solutions to portfolios
Practising reflexivity from activities to outcomes
Qualities of outcomes:
Desirability over multiple timeframes for different actors
Plausibility of contributing to the societal challenge at micro to macro system scales
networkedness of interventions within a portfolio
Using this framework, I assessed a intervention portfolio of seven intervention sets comprising 21 product-service systems resulting from a transition design process to reduce food waste. This evaluation was performed through a narrative-based study involving 312 participants from different actor groups (consumers, companies, and experts), enabling participants to envision and assess their potential within real-world contexts.
Outcomes
Transition Design Framework
Evaluation of 21 innovations
The outcomes reflect the interconnected nature of the framework’s activities and evaluative criteria:
Desirability: Interventions were perceived as desirable when they aligned with the needs and expectations of different actors. This alignment was crucial to fostering adoption and engagement.
Plausibility: The interventions demonstrated plausibility by addressing challenges across micro (individual), meso (organisational), and macro (systemic) levels. Their systemic alignment ensured relevance over time.
Networkedness: The portfolio approach underscored the importance of designing complementary and reinforcing interventions that collectively addressed systemic challenges.
By employing a narrative-based approach, the study revealed how key actors perceived the effectiveness of these interventions in fostering the desired transition. This evaluation enabled reflection not only on the interventions themselves but also on the processes that led to their development.
The statistical analysis confirmed that a transition design process, engaign gin the outlined activities successfully result in interventions that meet the needs of various stakeholders (consumers, companies, experts) and are likely to drive both individual and systemic behaviour changes towards food waste reduction
This work addresses a critical gap in the research and practice, where the relationship between design activities and outcomes is often fragmented. The framework operationalises this relationship, providing a tool to foster dialogue within transition design processes, facilitate collaboration, and communicate the value of transition design to stakeholders.
