The Null Institute | Independent research note | 13 November 2026
By Ilya Nikitin, PhD. Aggregate-data audit and sensitivity analysis; not peer reviewed. Original pupil-level and work-time records were not available.
Can AI tutors free up teachers’ time? A recent British experiment gives a more complicated answer than its impressive AI approval rate suggests. In a 10-week trial run by Google and the maths platform Eedi, specialist educators approved 94.7% of AI-drafted replies unchanged. That sounds efficient. But someone still had to check 94,385 drafts before pupils could see them.
We examined the 29 October 2026 technical report and independently recalculated the workflow counts, learning contrasts and illustrative supervision-time scenarios. The result is not that AI tutoring failed. It is that the published study cannot yet tell a school how many staff hours it will save, or whether the extra help produces better general maths attainment.
Two different measures of learning
The study assigned 1,179 pupils in ten UK state secondary schools to one of four kinds of support after a maths error: Eedi’s existing videos and hints; an expert human tutor; or one of two AI tutors supervised by a human. The control group received an established educational intervention, not nothing.
| Outcome | Existing support | Human-supervised AI | Interpretation |
|---|---|---|---|
| Correct on the next similar diagnostic question | 43.4% | 56.5–56.9% | +13.1 to +13.5 percentage points on an immediate outcome |
| Independent Star Maths score after ten weeks | Reference | −0.01 to +0.02 SD | No detected advantage over existing support |
The report’s 95% credible intervals for the AI effects on Star Maths were −0.15 to +0.14 and −0.12 to +0.16 standard deviations. This does not prove that AI has no longer-term benefit or is equivalent to a human tutor. The ranges still allow meaningful benefits and harms; ten weeks may also be too short to measure lasting changes.
The Eedi study update says 916 pupils had both initial and final Star Maths scores: 263 fewer than the 1,179 enrolled (22.3%). Those are missing paired test scores, not necessarily dropouts. With the available aggregates, we could not establish whether missingness differed by randomised group.
What the 95% approval rate does not tell you
Of 114,935 AI drafts generated, 20,550 were superseded before review. Human tutors evaluated 94,385, approving 89,406 unchanged and changing or regenerating 4,979. Those counts reconcile to the reported 94.7% approval rate.
Approving without editing still requires attention. The trial used 58 experienced maths educators who rotated between direct tutoring and supervising AI. The report did not measure review time or demonstrate a net saving in staff hours. We therefore calculated three hypothetical review-speed scenarios:
| Assumed attention per reviewed draft | Total over ten weeks |
|---|---|
| 5 seconds | 131 reviewer-hours |
| 15 seconds | 393 reviewer-hours |
| 30 seconds | 787 reviewer-hours |
These are scenarios, not observed hours or savings. They omit the additional work involved in rewriting messages, safeguarding and technical support. They do not compare with the time a human would need to tutor those same pupils directly. The study therefore cannot establish that this particular supervised setup is cheaper per pupil or that it releases school staff.
Deeper conversations, fewer topics
Pupils assigned to AI completed a median of 99 study units, versus 132 under existing support: 25% fewer. Those with human tutors completed 77 units. AI pupils also covered fewer distinct maths concepts, 57 versus 72 for standard support. The developers suggest that slower, deeper conversations may help later. That is plausible, but the trial has not yet demonstrated it.
Completed units were measured after randomisation. Adjusting models for this may explore mechanisms, but it changes the causal question and cannot prove that slower curriculum coverage caused the missing Star Maths improvement.
Safety requires careful wording
One retrospective audit identified six standalone maths or tone problems among the AI drafts, none of which reached pupils. A separate conversation-context audit identified seven other mathematical mistakes that did reach pupils. These statements describe different audits and are not contradictory. A low error rate is encouraging, but it does not mean the AI never gave an incorrect response.
What a school pilot should measure
Compare AI with the support pupils already receive, ideally with random assignment and independent maths assessments taken without AI. Record actual human minutes for reviewing, editing, safeguarding and technical support alongside staff training, subscriptions and cost per pupil. Track curriculum coverage, longer-term learning, and outcomes for pupils with SEND and other support needs. Without both a measured learning increment and a complete staff cost, claims of cost-effectiveness remain premature.
Other evidence does not justify a sweeping verdict for or against AI tutoring. A two-year Khanmigo trial in 18 US middle schools found small maths gains similar to those achieved with Khan Academy practice without AI. An independent EEF/NFER evaluation of adaptive Maths-Whizz in 63 English primary schools found an effect of +0.08 SD over a longer period. Different tools, pupil populations and comparison groups make direct pooling inappropriate.
Methods and reproduction
This is an independent recalculation of published aggregates, not a new experiment or a replication of the original Bayesian models. We did not have pupil-level outcomes or the timestamps needed to calculate actual human review time. Every review-hours scenario follows 94,385 × assumed seconds per check ÷ 3,600. The share missing paired test scores is (1,179 − 916) ÷ 1,179. The study-unit contrast is (99 ÷ 132 − 1) × 100. We checked the approval percentages against the authors’ original whole-number denominators.
Primary source: LearnLM Team, Google & Eedi (2026), technical report, 22 pages. Additional context: Eedi release; Oreopoulos & Low (2026); EEF/NFER Maths-Whizz trial.
Bottom line: Human-supervised AI produced substantial gains on immediate maths questions in this trial. An advantage on the independent broad test, and a net saving in human staffing time, remain unproven. These claims need separate tests.