How can scenario planning help evaluate the impact of research?

DECI-3 Blog

How can scenario planning help evaluate the impact of research?

By Ricardo Ramírez and Dal Brodhead, DECI-3.

A recent i2insights blog by Bonnie McBain outlines three aspects of scenario planning that help: define goals, design strategy, and produce useful and defensible projections.  Scenario planning is also useful when working to resolve uncertainties which are common to research efforts – especially those aimed at influencing decision-makers.  McBain emphasizes that in her opinion designing scenarios includes the definition of descriptive story telling with numerical modeling – a process that requires stakeholders to dialogue iteratively. Scenario planning resembles the process of definition of evaluation questions in contexts of uncertainty, and it overlaps with communication planning in wording those questions in language that is understood by different audiences.

McBain’s blog resonates with the DECI-3 project, as our partners are researchers seeking to influence policy. We, they, and our funder, are well aware of the difficulty of predicting policy outcomes given that effective policy change is so often emergent, accidental, and hard to replicate.   While McBain’s piece is a useful brief on the value of scenario planning, an article by James Derbyshire under the title “Use of scenario planning as a theory-driven evaluation tool”[1] brings this difficulty of predicting outcomes closer to home.

The article introduces what Derbyshire has termed ‘theory-driven evaluation’ (TDE) as an approach that describes an evaluation’s purpose, user and uses, activities, strategies and methods. It sounds very much like the utilization-focused evaluation approach that we are familiar with as part of the DECI-3 project.  However, among the challenges that Derbyshire identifies of TDE, is identifying change when faced with the emergent nature of the task at hand – something relevant in our context of evaluating leading edge research for policy initiatives.

The article explains how scenario planning can assist TDE by:

“(a) Making explicit initial causal logic and theory;

(b) Facilitating useful debate and discussion amongst multiple stakeholders; and

(c) Facilitating consideration of how contingent and complex causation may lead to unexpected outcomes…” (quoted from the abstract of Derbyshire, 2018)

The first point above is familiar to us in that TDE seeks to define (up-front) the logic of an intervention, especially its causal chain. In the DECI-3 context, we are supporting projects in outlining a given theory of change to illustrate the expected sequence of actions and changes, and to make the underlying assumptions about how change is expected to happen more explicit.   Derbyshire sets out his TDE through a six-stage process with scenario planning augmenting theory of change design. Table 1 below shows some parallels the steps in utilization-focused evaluation followed with the DECI-3 process[2]. For instance, both share the development of indicators down the road – in step 5 of 6.

The second point in the above list emphasizes stakeholder engagement as an iterative process to clarify what is doable, as well as how and through what actions.  Derbyshire admits that one the limitations of TDE is the actual process of stakeholder engagement, and especially how ownership over the process can be achieved.  This is particularly challenging in unchartered territory -such as in emergent research on policy arenas where the stakeholders may be unable to appreciate the essence of the research topic, let alone the convoluted process for bridging research-to-policy.

On his third point, which notes how complex causation may lead to unexpected outcomes; Derbyshire summarizes a number of steps of scenario planning. It adds value to our approach in its emphasis on making the effort to define the driving forces of change in the system and clustering them into a matrix. These forces refer to the contextual dimensions that are unlikely to be shifted in the short run and which may shape the outcomes of an intervention.  This clustering of forces provides a backdrop identifying plausible future outcomes.  It takes several iterations to establish the major forces, such as a gradient from political stability to instability; from economic development to economic stagnation; from effective regulated regimes to emergent ones. In the case of our research partners, possible clusters might include the differences between evidence-based policy making and ideology-based policy-making; and good Vs. poor alignment between research findings and policy windows.  The matrix typically focuses on two gradients, creating a 4-quadrant ‘uncertainty matrix’, where the different plausible outcomes can be located on the basis of where they lie across both gradients of contextual variables[3]. In the literature, there are examples of iterations where the definition of the gradients shifts as a clearer understanding emerged from the stakeholder interactions[4].

A challenge for many of our partners is stakeholder engagement, especially in emergent topics where the policy processes are complex and a project’s ‘convening capacity’ may be limited. What may be more realistic as an approach for us to consider in our next engagement is for the project leader to lead the scenario planning process and to consult with stakeholders in an opportunistic manner to verify and adjust the design of the intervention.  Beyond these practical challenges, scenario planning adds a methodology to the evaluation of projects undertaken highly un-predictable change, as well as an acknowledgment of their emergent nature. This advance appears to be new evaluation territory and we welcome feedback from readers with case studies from which we may learn together.

[1] Derbyshire, J. (2018).  Use of scenario planning as a theory-driven evaluation tool. Futures Foresight Sci.  https://doi.org/10.1002/ o2.1

[2] The DECI-3 hybrid outlines a set of decision-making steps in both evaluation and communication planning. In our experience, it challenges project managers to be clear on their strategy, and for this reason we have introduced theory of change into the process.

[3] The two-by-two matrix has been further challenged, see: https://www.researchgate.net/publication/259094631_Rethinking_the_2_2_scenario_method_Grid_or_frames

[4] Ramírez, Rafael.; Mukherjee, M.; Vezzoli S. & Matus Kramer, A. (2015). Scenarios as a scholarly methodology to produce “interesting research”. Futures 21: 70-87.