Teaching a Machine to Read the Room
There is a moment every programming instructor knows. A student has been staring at the same error for twelve minutes. Their keystrokes slow from a rhythm to occasional, reluctant taps. They have stopped asking questions. Then they open a new tab and type the name of a large language model.
The problem vanishes in seconds. And so does the learning.
This is the productive failure paradox — the uncomfortable truth at the centre of our research. The cognitive struggle that feels worst in the moment is often the mechanism by which understanding becomes durable. Frustration, within a window, is not an obstacle to learning. It is the process. AI coding assistants, in their eagerness to help, have become extraordinarily efficient at eliminating it.
Berry is our attempt to build something that knows the difference.
The Problem with Knowing the Answer
Learning science has been building a case for productive struggle for decades. Students who work through difficulty — who make errors, notice them, and repair them without external rescue — develop qualitatively different understanding from those who receive clean, correct solutions (Kapur, 2008).
The knowledge is more transferable. The mental models are sturdier. The competence is real rather than performed.
But there is a boundary condition that makes this complicated in practice. Productive struggle operates within a window. Before a certain threshold, confusion is a generative state — the mind is actively constructing. Past that threshold, confusion tips into defeat. The student does not learn from frustration that becomes disengagement. They simply disappear.
The Design Problem
The challenge is not whether to intervene — it is when.
Too much support, too early, collapses the productive struggle that builds real understanding. Too little support, too late, loses the learner entirely. The window between those two failures is narrow, dynamic, and different for every student on every problem.
Most intelligent tutoring systems navigate this through performance signals: correctness rates, time-on-task, error frequency. These are useful proxies but they are, fundamentally, measurements of what has already happened. They are retrospective.
Berry is designed around a different premise:
The most actionable signal is not what the student has done, but what they are feeling right now.
Two Layers of Listening
Detecting emotion in a programming learning environment is a harder problem than it first appears. Students do not narrate their inner states. They do not say “I am now entering a state of productive confusion.”
They type, compile, pause, delete, rephrase — and occasionally send a message stripped of context.
The signal is distributed across modalities and often deliberately understated.
Berry addresses this through a two-layer detection architecture, where each layer captures a fundamentally different kind of signal — and the two are fused rather than treated as alternatives.
🧠 Berry — Emotion Detection Architecture
Layer 1: Behavioural Signal
Continuous, passive observation of how a student interacts with the environment over time — the rhythm and texture of engagement rather than its explicit content.
This layer asks: Does the pattern of activity look like someone who is thinking, or someone who has stopped?
Layer 2: Semantic Signal
Active analysis of the language a student uses when they do choose to communicate — including the vocabulary of frustration, confusion, and disengagement specific to programming contexts.
General-purpose sentiment tools miss most of this. Berry uses a purpose-built lexical layer calibrated to the domain.
EMP Engine: Orchestration
The fused emotional signal feeds a decision function that selects a pedagogical strategy — not just whether to intervene, but how:
- Scaffolding level
- Delivery mode
- Degree of directness
All calibrated to this student, at this moment, on this problem.
A Key Design Choice
Emotion is not a secondary metric.
It is the primary signal.
- Cognitive state → informs what to say
- Emotional state → determines whether to intervene and how
This inversion distinguishes Berry from most adaptive learning systems.
The Orchestration Layer
Between detecting emotion and deciding what to do lies a complex decision space:
- Syntax frustration vs conceptual misunderstanding
- 40 seconds vs 4 minutes of confusion
- Different student preferences for scaffolding
Emotion tells you the window is open.
Intelligence is knowing what to put through it.
Berry’s Emotion-Mediated Pedagogical (EMP) framework integrates:
- Emotional state
- Cognitive state
- Interaction history
It selects:
- Strategy (question, hint, example, etc.)
- Delivery (directness, abstraction level)
⚠️ Critical constraint:
Full code disclosure is never allowed.
The goal is always:
Return the student to productive agency — not remove the problem.
The Scaffolding Challenge
One of the hardest problems:
Where does scaffolding end and solution-giving begin?
This is not a threshold.
It is a moving gradient.
Dependent on:
- Student
- Problem
- Moment
Maintaining this boundary is still an open research challenge.
Six Iterations — What We Learned
Berry is not a product. It is a research artefact built through Design-Based Research (DBR).
Early Iterations
- Established core premise: emotion can be detected
- Used simple thresholds
- Learned: thresholds are often wrong
Middle Iterations
- Introduced two-layer architecture
- Built domain-specific lexical model
- Learned: general sentiment tools are structurally inadequate
Recent Iterations
- Formalised EMP framework
- Defined productive struggle window
- Identified scaffolding vs solution as core challenge
Current Deployment
- Running at Toronto Metropolitan University
- Testing real impact on learning outcomes
- Generating empirical validation
What We Are Trying to Learn
1. Detection Reliability
Can emotion detection work in real environments (not just labs)?
2. Strategy Effectiveness
Which interventions actually change student trajectories?
3. Learning Outcomes
Does this improve measurable learning results?
4. Design Principles
What generalizable principles emerge from six iterations?
The Deeper Question
As AI becomes ubiquitous in learning:
Will the ability to use AI be mistaken for the ability to think?
Berry exists to resist that substitution.
An Open Invitation
Berry is currently deployed at Toronto Metropolitan University across programming courses.
We are exploring:
- Assessment integration
- Academic integrity implications
- Ethics of adaptive support
We are also considering:
- Research collaborations
- Shared infrastructure
- Open-source directions
If you are:
- A CS educator
- A learning scientist
- An institution exploring similar problems
We would like to hear from you.