Method

The synthetic case format, the supervised training recipe, the evaluation method, and the scope of the recorded results.

1. What a Case Is

A case is one teaching moment: a mathematics question, what a learner did and said about it (a synthetic learner situation), the decision the tutor must make, one good reply, and a checklist of short yes or no lines that define a good reply. The failure the cases target is specific: when a learner states an incorrect way of working with fractions or decimals, a tutor either does not name the learner's likely error, or states its diagnosis as certain rather than likely.

The synthetic questions cover grades 4 and 5: fractions, decimals and place value. Cases and training examples are kept in groups by question, and a group is either wholly available for training or wholly kept for testing, so no evaluation question appears in the training file.

2. How a Reply Is Graded

Two graders, one Claude model and one GPT model, each mark every line of the case's checklist for a reply, alone and without knowing which tutor wrote it.

RuleIn plain words
A line is met only if both graders say yesThe stricter reading; a line one grader doubts does not count
A serious error counts if either grader sees itA serious error (for example, confirming a false claim) sets the score to 0
Score: lines met divided by linesBetween 0 and 1
A case passes when every line is metAnd no serious error is seen

The Duel board sorts its cases by how hard they are to judge: the share of checklist marks, over every tutor's reply to a case, on which the two graders disagreed, cut into thirds (easy, moderate, difficult). It is fixed by the graders' marks before anyone plays and does not depend on which tutor wins.

Two checks make a score readable. The ceiling check grades the reference replies like any tutor's: they score 0.99, so the test can be passed and the graders can see it. The replicate trains the same tutor twice: the two runs differ by about 0.02, which describes these observed replicates and is not a universal significance threshold.

3. How a Tutor Is Trained

Supervised fine-tuning (SFT). The tutor reads many training examples as conversations (the instruction, the question with the learner's move, and the reference reply) and learns to reply the same way. Training changes a small adapter beside the model (LoRA, rank 32), for 8 passes over the data at a learning rate of 0.0003.

4. The Tutors Compared

  • Untuned: the open model as released (Qwen3-8B, Qwen3-14B and Qwen3.8-27B).
  • Trained on our data: the same model after SFT on 502 to 1,232 training examples.
  • Trained on generic replies: the same model after SFT on the same prompts answered with correct, clear, generic tutoring that never diagnoses the learner. This is the comparison that shows whether our examples teach anything beyond the format of a tutor's reply.
  • Examples in the prompt: the untuned model with five of our training examples pasted into its prompt, no training.
  • General-purpose models: GPT-6 Sol, Kimi K3 and Nemotron 3 Ultra as released, on the same tutor prompt, with no training on our data. They are compared with our best tutor blind on the development cases, by the two graders and by a third judge from a third company (GLM 5.3), and graded on the 76 test cases like every tutor.

5. Evaluation Rules

Run plans, recorded internally before each run, specify the run questions, model configurations and decision rules used to assess the results. The 76 cases were excluded from training but examined across a sequence of experiments. Because of that repeated use, they serve as historical evaluation cases rather than a fresh test set. The Duel board shows them, with every tutor's recorded reply and both graders' marks.

6. Scope of the Results

The cases and training examples are synthetic and the replies are graded by models. The results show that a tutor trained on these examples learns the teaching moves their checklists define, by a margin far larger than the noise between runs, and that a generic tutor trained on the same prompts does not. They do not show that a learner learns more from the trained tutor; that would need a study with learners.

Close window

Contact Us

20 to 6,000 characters.

Close window Close window