Learning through decisions - what the research says about scenario-based design

This is the second post in a series exploring the evidence behind learning design decisions. The first post examined why the tell-and-test format underperforms and what a century of retrieval practice research tells us about retention. This post builds on that foundation by looking at a different question: what happens when we ask learners to make decisions rather than absorb information?

In the previous post, we established that retrieval practice - being asked to recall or apply information - consistently outperforms re-reading and passive exposure. The testing effect is one of the most reliably replicated findings in educational psychology, with effect sizes around g = 0.50 across hundreds of studies.

But retrieval practice has a limit. Most of the evidence behind it involves recalling facts, concepts, and procedures - the kind of knowledge that can be tested with a question and a correct answer. Workplace learning often demands something different: the ability to read a situation, weigh competing considerations, and act appropriately when the textbook answer is not clearly labelled. That is where scenario-based learning enters - and where the research gets genuinely interesting.

The problem with knowing without being able to use

There is a well-documented gap in learning research between declarative knowledge - knowing that something is true - and procedural or applied knowledge - knowing how and when to use it. The gap shows up most clearly when learners who can answer factual questions correctly still fail to apply the underlying principles in unfamiliar situations.

This is not a failure of motivation or attention. It reflects something fundamental about how knowledge is stored and retrieved. Information learned in one context - a course, a slide, a definition - is not automatically available in a different context, like a conversation with a difficult client or an ambiguous ethical situation at work. Cognitive psychologists call this the transfer problem, and it is the central challenge of training design.

Scenario-based learning addresses the transfer problem directly. Rather than teaching principles and then hoping learners will apply them, it places learners inside a situation that demands application. The sequence is reversed: encounter the problem first, work through it, then surface the principle. The learning is anchored to a context that resembles the one where it will actually be needed.

What the meta-analyses actually show

The research on problem-based and scenario-based learning is broadly positive but contains important nuance that is worth understanding before reaching for it as a design default.

Dochy, Segers, Van den Bossche, and Gijbels (2003, Learning and Instruction) conducted a meta-analysis of 43 studies comparing problem-based learning with conventional instruction. Their summary has become one of the most quoted - and most misquoted - findings in instructional design: students in problem-based conditions showed robust positive effects on skills and knowledge application, with no single study reporting a negative result, while showing a slight negative effect on factual knowledge recall.

The misquotation usually stops at the negative finding. The full picture is more interesting: students in problem-based conditions gained slightly less knowledge in the short term, but retained significantly more of what they did gain over time. As Dochy and colleagues put it, they "gained slightly less knowledge, but remember more of the acquired knowledge."

Strobel and van Barneveld (2009) synthesised eight prior meta-analyses and confirmed the pattern. Problem-based learning was superior for long-term retention, skill development, and learner satisfaction. Traditional instruction was superior for short-term retention on standardised tests. If your success metric is a knowledge check at the end of a course, conventional instruction may win. If your metric is behaviour six months later, the picture reverses.

Gijbels, Dochy, Van den Bossche, and Segers (2005) added an important refinement: the advantage of problem-based approaches grew largest when assessments measured understanding of principles linking concepts rather than recall of isolated facts. This is the kind of understanding that actually predicts performance in complex roles.

The productive failure finding

The most counterintuitive - and practically useful - finding in this space comes from Manu Kapur's research programme on what he calls productive failure.

The central claim is this: asking learners to attempt a problem before receiving instruction produces better learning than providing instruction first and then asking learners to practise. Not because they solve the problem correctly - most do not. But because the struggle activates prior knowledge, surfaces the gaps in existing understanding, and creates a kind of cognitive readiness that makes the subsequent instruction far more effective.

Sinha and Kapur's (2021, Review of Educational Research) meta-analysis synthesised 53 studies involving over 12,000 participants. Problem-solving before instruction outperformed instruction before problem-solving at g = 0.36 overall, rising to g = 0.58 with high-fidelity implementation of the productive failure design. After adjusting for publication bias, the effect for conceptual knowledge and transfer reached g = 0.87. Crucially, productive failure produced no compromise on procedural knowledge - learners matched direct instruction on procedural tasks while substantially exceeding them on conceptual understanding and transfer.

For scenario-based e-learning, this has a direct design implication. The instinct is usually to front-load information - teach the concept, then present the scenario as practice. The evidence suggests inverting this, at least for conceptual learning: present the scenario first, let learners attempt it with what they already know, then use the debrief to introduce the principle. The difficulty is not a design flaw. It is the mechanism.

What makes a scenario actually work

Not all scenarios are equal, and the research helps explain why some produce genuine learning while others feel like window dressing on a multiple-choice question.

The critical variable is decision authenticity. Scenarios that present obviously correct answers alongside clearly wrong distractors require recognition, not reasoning. They test whether learners can identify the right answer when it is labelled by context - which is a very different skill from navigating a genuinely ambiguous situation where reasonable people might disagree.

Chi and Wylie's ICAP framework, discussed in the previous post, is useful here. A scenario that asks learners to choose between "report the issue immediately" and "ignore it and hope it resolves" is operating in the Active mode at best - learners are doing something, but not necessarily thinking. A scenario that presents a situation where reporting has costs, not reporting has risks, the right authority is unclear, and time pressure is real - that forces Constructive engagement. Learners must generate a position, not recognise one.

Several design features are supported by evidence as genuinely meaningful rather than cosmetic.

Consequences matter more than correctness. Evans and Gibbons (2007, Computers & Education) found that interactive e-learning significantly improved transfer and problem-solving but showed no significant advantage on factual recall. The mechanism was consequential feedback - seeing what their decision caused - rather than being told they were right or wrong. Consequences create the emotional and cognitive trace that makes the scenario memorable and applicable. A wrong answer followed by a natural consequence teaches more than a wrong answer followed by "incorrect - try again."

Branching adds value when it reflects real complexity. Branching scenarios that allow different paths to produce genuinely different outcomes reflect the actual structure of complex decisions. Branching that reconverges after one or two steps - where the story resets regardless of what the learner chose - provides the appearance of consequence without the substance. Learners notice.

Characters and context reduce abstraction. There is consistent evidence that grounding problems in realistic characters and contexts aids transfer compared to abstract case descriptions. This is one mechanism behind the spacing and interleaving effects: context-rich scenarios help learners build flexible representations of when and where to apply a principle, rather than a single rigid association between a cue and an answer.

The important counterargument

In 2006, Kirschner, Sweller, and Clark published an influential paper arguing that minimally guided instruction - which includes discovery learning and many constructivist approaches - consistently underperforms explicit instruction for novice learners. The paper has over 4,000 citations and is regularly used to argue against scenario-based approaches.

The argument deserves engagement rather than dismissal. Cognitive load theory, which underpins it, is well-supported. Novice learners lack the schemas that allow them to distinguish relevant from irrelevant information in a complex scenario. Without those schemas, the cognitive load of navigating an unfamiliar situation can overwhelm working memory, leaving little capacity for actual learning. The result is busy-feeling activity with limited retention.

The productive resolution, developed by Hmelo-Silver, Duncan, and Chinn (2007) and supported by the productive failure evidence, is that the dichotomy between guided and unguided instruction is false. Well-designed scenario-based learning is not minimally guided - it is specifically guided. The scaffolding is embedded in the scenario structure itself: the situation is complex but bounded, the decision points are clear, the consequences are informative, and the debrief provides the explicit instruction that connects the experience to the underlying principle.

The expertise reversal effect adds a further refinement: the amount of scaffolding needed decreases as expertise grows. A scenario that appropriately challenges a new compliance officer will bore a senior one. Designing for a single learner profile is always a compromise - and the cost of that compromise falls on whoever is furthest from the assumed baseline.

What this suggests about design

The research does not suggest that every piece of e-learning should be a scenario. Factual knowledge - definitions, procedures, regulatory requirements - can be taught efficiently through direct instruction, and adding a scenario wrapper adds development time without necessarily improving outcomes for straightforward content.

Where scenario-based learning earns its complexity is exactly the territory where tell-and-test fails: situations requiring judgement, situations where the stakes of a wrong decision are real, situations where the goal is not just remembering a rule but knowing when and how to apply it under pressure.

A few principles follow from the evidence.

Present the problem before the principle, at least some of the time. Let learners struggle with a situation using what they already know, then use the debrief to introduce or reinforce the underlying concept. The struggle is not a problem to be designed away - it is doing some of the learning.

Design for authentic decisions, not recognisable answers. If a reasonably attentive learner can identify the correct path without engaging with the substance of the scenario, the scenario is not doing its job. The decision should require genuine reasoning about the specific situation, not pattern-matching to "what does a compliance course want me to say?"

Use consequences rather than judgements as feedback. Showing learners what their decision causes - in the logic of the scenario - teaches more than telling them they were right or wrong. The consequence anchors the learning to the decision rather than to the assessment.

And match scenario complexity to learner expertise. What challenges a novice overwhelms a beginner and bores an expert. A single scenario designed for the average learner is serving no one especially well.

A note on AI and scenarios

One practical constraint on scenario-based learning has always been development time. A branching scenario with multiple paths, realistic characters, and meaningful consequences takes considerably longer to build than a slide-and-test format. This is a real cost, and for many organisations it is the reason tell-and-test persists despite its limitations - not ignorance of the research, but production economics.

AI-assisted authoring changes that calculation in ways that are still being worked out. Generating plausible character dialogue, drafting decision branches, and creating consequence text are exactly the kinds of tasks where AI can accelerate production without compromising the instructional logic - provided the instructional logic is supplied by the designer. The research on what makes scenarios effective is not something AI can substitute for. But the time cost of building out scenarios once the logic is established is substantially lower than it was five years ago.

The next post in this series will examine a finding that most e-learning design ignores entirely: the expertise reversal effect, and what it means for courses designed for mixed-experience audiences.

Sources: Dochy, Segers, Van den Bossche & Gijbels (2003); Strobel & van Barneveld (2009); Gijbels, Dochy, Van den Bossche & Segers (2005); Sinha & Kapur (2021); Chi & Wylie (2014); Evans & Gibbons (2007); Kirschner, Sweller & Clark (2006); Hmelo-Silver, Duncan & Chinn (2007).