Teaching Models to Read and Respond to a Learner's Reasoning

Abstract

When a learner makes a mistake with fractions or decimals, the reasoning they write down usually shows why. We ask whether a language model can use that evidence. Can it name the likely error, present it as a likely explanation and not a certainty, credit what is correct, explain the idea and end with a next step? SENSUS Bench v1.0 tests this on a development set of 76 grade 4 and 5 cases. Two model graders from different companies score each reply line by line against a checklist written for its case. We release SENSUS SFT, 1,015 synthetic training conversations, and use it to fine-tune low-rank adapters on three open models. Qwen3.8-27B trained on SENSUS scores 0.825 on average over two training runs (0.811 and 0.838), against 0.290 for the same base before training, a pooled gain of +0.534 (95% CI [+0.476, +0.591]). GPT-6 Sol, the highest-scoring general-purpose model, scores 0.525. A control trained on generic tutor replies to the same prompts scores 0.445, so the SENSUS replies as a whole teach far more than fine-tuning on tutor replies in general. The general-purpose models received a one-line tutor instruction that does not describe the target behaviours, and most of their gap lies in those behaviours. With the length, hedging and certainty rules set aside, the fine-tuned 27B model is level with Opus 5.5 and still ahead of GPT-6 Sol. On these cases the share of 27B replies that both graders marked free of false statements rose from 75% untuned to 91% for the second SENSUS run, and replies triggering a listed serious error fell from 26 of 76 to 0. GPT-6 Sol still made no false statements, and the smaller fine-tuned models make them more often than any general-purpose model.

1. Introduction

Ask a capable language model to help a ten-year-old who has decided that 4/10 is bigger than 3/6 because 4 is bigger than 3, and it will usually explain how to compare fractions correctly. It is less likely to say what went wrong in the child's own method. Feedback research suggests that this second step matters. Feedback works best when it addresses the gap between what the learner did and the goal (Hattie and Timperley, 2007), and it should come in pieces small enough to act on (Shute, 2008). Errors in elementary fractions and decimals are often systematic. Learners carry whole-number rules into fractions (Ni and Zhou, 2005) or judge a decimal by how many digits it has (Resnick et al., 1989; Steinle and Stacey, 2004). Learning research has long treated such errors as knowledge in transition (Smith et al., 1993).

SENSUS targets two specific failures. When a learner states an incorrect way of working, a model often does not name the learner's likely error, or it states its diagnosis as certain when it can only be likely. We define five behaviours for a reply to a learner's written reasoning. The first two address those failures directly, and the other three make the reply useful.

1. B1. Name a likely error tied to what the learner wrote.
2. B2. Hedge the diagnosis. The cause of an error is an inference about someone else's thinking.
3. B3. Credit what is correct.
4. B4. Explain or model the idea the error concerns.
5. B5. End with a next step the learner can take, usually a question.

A reply should also be correct and short. We drew these behaviours from research on feedback, misconceptions and responsive teaching (Section 3.3), and this study does not test whether each one, or its weight in the score, improves learning. SENSUS Bench turns them into case-specific checklist lines that measure observable properties of a single reply, and SENSUS SFT gives 1,015 examples of replies that show them. Figure 1 shows one case from start to finish.

We fine-tune three open models on SENSUS SFT and evaluate them, their untuned bases, matched generic controls and four general-purpose models on SENSUS Bench. Five findings stand out.

1. About a thousand examples teach the behaviour. Qwen3.8-27B rises from 0.290 to a two-run mean of 0.825 (0.811 and 0.838; pooled gain +0.534, 95% CI [+0.476, +0.591]). A control trained on generic tutor replies to the same prompts reaches 0.445, so the SENSUS replies as a package, not tutor-style fine-tuning in general, account for the gain.
2. A 27B open model outscores every general-purpose model we tested. It leads on all 12 standards and beats GPT-6 Sol on 68 / 3 / 5 cases (wins / losses / ties), under the same one-line tutor instruction (Figure 2).
3. The 27B model also became more accurate. The share of replies both graders marked free of false statements rose from 75% untuned to 91% for the second SENSUS run, and replies with a listed serious error fell from 26 of 76 to 0.
4. The gain lies in the targeted behaviours. Fine-tuning moves naming, hedging and the next step (Figure 4). With the length, hedging and certainty rules set aside, the fine-tuned 27B model is level with Opus 5.5 and still ahead of GPT-6 Sol.
5. Style can outrun accuracy in smaller models. At 8B and 14B the fine-tuned models adopt the reply style fully but state something false in 51% and 30% of replies.

The general-purpose models were not told about B1 to B5, so these comparisons describe their default behaviour under a short tutor instruction. Prompting alone did little in the one recorded test: Qwen3-8B given five SENSUS examples in its prompt scored 0.312, against 0.239 untuned and 0.578 to 0.647 after fine-tuning. A general-purpose model prompted with B1 to B5 was not run, and it is the first item of future work.

The paper makes four contributions.

1. SENSUS Bench v1.0. 76 cases over 12 Common Core standards (NGA and CCSSO, 2010), with 549 checklist lines and 218 serious errors. The two graders agree on 94.7% of line judgments (κ = 0.89).
2. SENSUS SFT. 1,015 conversations under CC BY 4.0. No question is shared with the benchmark, and the release includes checksums and an offline verifier.
3. Training results. Fine-tuning raises Qwen3.8-27B from 0.290 to a two-run mean of 0.825 (pooled gain +0.534, 95% CI [+0.476, +0.591]), with matched generic controls and two training runs per recipe at 8B, 14B and 27B. At 27B, false statements fall from 25% to 9% of replies and serious errors from 26 to 0.
4. A breakdown of the gain. Rescoring the recorded marks under seven rules separates the behavioural gain from mathematical content and shows what fine-tuning cost in accuracy.

2. Related Work

Tutoring benchmarks. MathDial collects teacher moves in dialogues with a model that simulates student errors (Macina et al., 2023). Bridge records the decisions expert tutors make when a student errs, and shows that models improve when given those decisions (Wang et al., 2024a). The taxonomy of Maurya et al. (2025), which includes mistake identification and providing guidance, underlies the BEA 2025 shared task (Kochmar et al., 2025). MathTutorBench (Macina et al., 2025), stepwise verification before remediation (Daheim et al., 2024) and evaluation programs for educational assistants (Jurenka et al., 2024) cover broader tutoring skills, and TutorBench scores explanations, feedback and hints against rubrics written for each sample (Srinivasa et al., 2025). SENSUS is narrower. Its replies are single turns, and its target is a hedged diagnosis drawn from the learner's own words. Case-specific rubrics are not new. Our contribution is to pair them with this reply policy, an open training set, same-base training comparisons and an analysis of how the scoring rules shape the results.

Misconception diagnosis. Work on diagnosing misconceptions with language models has mostly addressed algebra (Otero et al., 2025), and models have been used to simulate students who hold specific misconceptions (Sonkar et al., 2024). We study fractions and decimals, and we ask the model to act on its diagnosis in the reply.

Fine-tuning tutors on synthetic data. TutorChat and SocraticLM fine-tune on large synthetic tutoring corpora (Chevalier et al., 2024; Liu et al., 2024), and Petukhova et al. (2026) add preference optimization for mistake remediation. Synthetic instruction data is now common practice (Wang et al., 2023; Long et al., 2024). LIMA showed that around a thousand examples can set a model's response style (Zhou et al., 2023), and imitation data can change style more than capability (Gudibande et al., 2024). Our results are consistent with both. We train low-rank adapters (Hu et al., 2022), which tend to forget less of the base model than full fine-tuning does (Biderman et al., 2024).

Feedback and teaching research. Research on feedback (Hattie and Timperley, 2007; Shute, 2008; Kluger and DeNisi, 1996), formative assessment (Black and Wiliam, 1998), student errors (Brown and Burton, 1978) and teacher noticing (Jacobs et al., 2010) shaped the reply policy. Section 3.3 sets out how each behaviour and checklist line follows from it.

Model graders. Model judges approximate human preferences in many settings (Zheng et al., 2023; Chiang and Lee, 2023). They are also sensitive to answer order (Wang et al., 2024b), favor their own outputs (Panickssery et al., 2024), and respond to length and style (Dubois et al., 2024; Feuer et al., 2025). Binary checklists make judgments more reliable (Lee et al., 2025; Ye et al., 2024), and conversation-specific rubrics with penalized criteria have been used at scale in health (Arora et al., 2025). We use both ideas, and Sections 7 and 8 measure how far our results depend on style.

3. SENSUS Bench v1.0

3.1 Cases

Each case is one teaching moment. It has a mathematics question labeled with its grade band and Common Core standard, a short description of what a synthetic learner did and said, the decision the reply must make, one reference reply, and a checklist of yes-or-no lines that define a good reply. The 76 cases cover 12 standards in grade 4 fractions and grade 5 place value and decimals (Tables 1 and 7). The model sees the system message “You are a mathematics tutor. A student is working on the question below. Reply to the student directly, as you would in the conversation.”, then a user message with the question and the learner's work, and replies once. Appendix B gives two cases in full.

3.2 Checklists and Serious Errors

Every case has 7 to 8 binary checklist lines. The first line requires that the reply say nothing false. One line asks for a hedged diagnosis (B2), one for a next step (B5), and one for about 120 words or fewer, following the advice to give feedback in manageable units (Shute, 2008). The remaining lines are about the case itself. A line may require naming the error of comparing numerators and denominators as separate whole numbers, for example, or showing that 4/10 is less than 3/6. Each case also lists 2 to 3 serious errors. Every case counts a diagnosis stated as certain as a serious error, and about two thirds also list endorsing a wrong answer. The word limit and the certainty rule are design choices that encode brevity and B2, and Section 7 reports results without them. A listed serious error and a false statement are scored differently. A reply can state something false without triggering any listed serious error; it then loses the correctness line but keeps partial credit, as Case 1 in Appendix B shows. We therefore report correctness separately and test a stricter rule in Section 7. Table 2 summarises what the checklist rewards.

3.3 Why the Checklist Asks for These Behaviours

We wrote the behaviours, and the checklist lines that test them, from research on feedback, mathematical misconceptions and responsive teaching. Mathematics education describes a teacher's response to a student's work as professional noticing: attending to the student's strategy, interpreting what it shows about their understanding, and deciding how to respond (Jacobs et al., 2010). B1 and B2 are the interpreting step, and B3 to B5 are the response. The checklist keeps only what can be seen in one written reply. Table 2 lists the main sources for each element.

Naming the likely error (B1). Errors in fractions and decimals tend to be systematic (Section 1). Brown and Burton (1978) modelled such errors as consistent faults, or bugs, in an otherwise working procedure, which is why the cause of an error can often be read from the work. Tutors do not always do this: in two samples of one-to-one sessions with non-expert tutors, the tutors worked from a rough sense of what the student knew and rarely diagnosed specific errors (Graesser et al., 1995). Research also suggests that studying and comparing incorrect and correct examples can support learning: students who compared incorrect and correct decimal examples, with prompts, gained more understanding and kept fewer misconceptions than students who studied correct examples only (Durkin and Rittle-Johnson, 2012). That study tested worked examples, not a tutor naming a learner's error; B1 applies its motivation by asking the reply to identify the specific likely error, tied to what the learner wrote, rather than restate the correct rule.

Hedging the diagnosis (B2). A wrong answer does not settle its cause, since different faulty procedures can produce the same answer (Brown and Burton, 1978). A diagnosis drawn from one written explanation is an inference, and B2 asks the reply to present it as one (“it looks like you may be…”). Stating the cause tentatively lets the learner confirm or correct it, which keeps the exchange formative (Black and Wiliam, 1998). It also keeps the feedback on the task rather than on the learner; in a large review of feedback studies, feedback that drew attention to the self tended to lower performance (Kluger and DeNisi, 1996). We know of no study showing that hedged wording itself improves learning. B2 is our design choice built on these findings, and Section 7 reports scores without it.

Crediting what is correct (B3). Productive teaching builds on what the student already thinks rather than replacing it (Leatham et al., 2015), and misconceptions usually grow out of knowledge that is partly right (Smith et al., 1993). Feedback is most useful when it addresses the task and the process the student used (Hattie and Timperley, 2007). B3 lines therefore require a specific correct step, such as the learner's correct renaming of a fraction; a general “Good job” is marked no.

Explaining the idea (B4). Conceptual understanding and procedural skill develop together, each supporting the other, as shown with grade 5 and 6 students learning decimals (Rittle-Johnson et al., 2001). The U.S. Department of Education's practice guide on fractions recommends measurement representations such as number lines and explaining why procedures work (Siegler et al., 2010). B4 lines ask for a reason a grade 4 or 5 learner can check, such as comparing to one half, a drawing or a common denominator. Stating which number is bigger does not count.

Ending with a next step (B5) and keeping it short. Feedback should tell the learner where to go next (Hattie and Timperley, 2007), and prompting learners to explain their own reasoning improves understanding (Chi et al., 1994). B5 asks the reply to end with a question or task the learner can answer; a trailing “Does that help?” alone does not count. The length line follows the advice to give feedback in manageable units (Shute, 2008).

Correctness and serious errors. Feedback does not help automatically. In the same review, more than a third of feedback interventions lowered performance (Kluger and DeNisi, 1996), which is why the checklist treats some replies as failures rather than weaker successes. A reply that says something false fails the correctness line. A reply that endorses a wrong answer or states the diagnosis as certain commits a serious error and scores zero. The zero is our scoring choice, not something the literature establishes: we treat confirming a learner's error and asserting a guess about their thinking as the two moves most likely to mislead them. Under this rule a confident diagnosis scores zero while some false mathematical statements keep partial credit, and the relative severity of the two has not been validated. Section 7 rescores the replies without the certainty rule and with a stricter correctness rule.

This research comes mostly from human teachers working with students over time. The checklist measures properties of a single written reply. Within a case, its lines count equally when no serious error is flagged, and a serious-error flag overrides them and sets the score to zero. This study does not test whether a reply that meets it helps a learner learn more (Section 10).

3.4 Metrics

For a reply to case c with n_c checklist lines, let m_c be the number of lines that both graders mark as met. The reply's score is

score_c = 0 if either grader marks any serious error, and m_c / n_c otherwise.

We report four quantities for each model. Mean score is the mean of score_c over the 76 cases, so every case counts equally whether it has seven or eight lines. Full passes counts cases with score_c = 1, meaning every line met and no serious error. Serious errors counts replies in which either grader flags at least one listed serious error. No false statement is the share of replies whose correctness line both graders mark as met. For pairwise preference judgments, net preference is (wins − losses) / resolved comparisons, where ties count as resolved. The reference reply for each case scores 0.991. That shows the checklists can be satisfied. The references were prepared alongside the checklists, so this is not independent evidence that the checklists are valid.

4. Training Data and Fine-Tuning

4.1 SENSUS SFT

SENSUS SFT v1.1 has 1,015 synthetic conversations over 884 distinct questions. Each conversation has the benchmark's system message, a user message with the question and the learner's work, and the reference reply as the assistant turn. 98% of the replies contain a hedging phrase, 97% end with a question, and none is longer than 109 words (Table 13). Cases are kept in groups by question, and each group goes wholly to training or wholly to evaluation. No evaluation question matches a training question exactly, though our checks do not rule out near-duplicates (Section 4.3). The benchmark and the training set still share their format, standards and kinds of misconception by design, so SENSUS measures in-distribution learning of a behaviour.

4.2 Generic Control

A control set answers the same prompts with 1,012 correct, clear, generic tutoring replies that never diagnose the learner. Its replies are short like the SENSUS replies, so training on it shows what fine-tuning on tutor-shaped text achieves without the diagnosis. Because it leaves out B1 and B2, generic training also pushes those behaviours below the untuned base (Figure 4). The control differs from SENSUS in several behaviours at once, so the comparison shows the effect of the whole SENSUS policy and not of any single behaviour. We therefore report the comparison with the untuned base alongside the comparison with the control.

4.3 Overlap with the Benchmark

No benchmark question appears in the training file (0 exact matches after normalizing case and punctuation). Questions in this domain share stock phrasing. 26 of 76 benchmark questions share at least one word 8-gram with some training question, and the largest share of one question's 8-grams found in a single training question is 33%. We compared questions only. Learner descriptions, replies and semantic similarity were not checked, so these results do not rule out near-duplicates in the full examples, which falls short of the contamination checks Sainz et al. (2023) recommend.

4.4 Recipe

Each model is trained with supervised fine-tuning on the conversations. It reads the instruction, the question and the learner's work, and learns to produce the reference reply. Training changes a rank-32 low-rank adapter beside the base model for eight passes over the data at a learning rate of 0.0003 (Table 3).

Eight epochs on about a thousand rows is a strong setting, and the fine-tuned replies are formulaic as a result. 97% of the 27B model's replies contain a hedging phrase, and a single two-word opening begins 24% of them. This is intended, but it is also a reason to check that the models keep their general abilities, which we did not measure.

5. Evaluation Setup

5.1 Models

We evaluate the fine-tuned models, the untuned bases, the generic controls and four general-purpose models (Table 4). Every model receives the same system and user messages. None sees the checklist, and none is told about B1 to B5. Each general-purpose model was evaluated once. Inference settings are not identical across model families (Table 4), and those differences, together with differences in model generation, limit comparisons across families. Unless stated otherwise, fine-tuned results use the second, latest completed run of each recipe.

5.2 Graders

Grader 1 is Opus 5.5 and grader 2 is GPT-6.1 Sol at medium reasoning effort. Each grader works alone. It sees the case as the model saw it, the checklist, the serious errors and one reply, with no indication of which model wrote it, and marks every line true or false (Appendix A). Each reply was graded once. Two overlaps between graders and graded models remain. Grader 2 comes from the same provider as GPT-6 Sol. For replies written by Opus 5.5, an alternate first grader from the same provider replaced Opus 5.5, which reduces the overlap without removing it. Section 8.2 reports each grader separately.

5.3 Statistics

Intervals on means are 95% percentile bootstrap intervals with 10,000 resamples (Efron and Tibshirani, 1993), and intervals on pass rates are Wilson intervals. Differences between models resample cases jointly. Because cases cluster by standard, Table 6 also resamples whole standards. Case-level wins and losses use a two-sided exact sign test that excludes ties, with Holm correction across the comparisons in that table (Holm, 1979; Dror et al., 2018). Grader agreement is reported as raw agreement and Cohen's kappa (Cohen, 1960). These intervals cover sampling over cases only, and they do not account for the recipe having been adapted to these cases (Section 6.3). They leave out variation between training runs, which was 0.014 to 0.028 between the two runs of each recipe, and variation in the graders. With 76 cases, only large differences are reliable (Card et al., 2020). Each run's questions, model configurations and decision rules were recorded before it started, and the 27B experiment named 27B against 14B as its primary comparison. We call this the pre-run primary comparison. It was recorded internally and not in a public registry, and because the cases had been used in earlier rounds, recording it does not make them a fresh test.

6. Results

6.1 Main Results

All three fine-tuned models score above every general-purpose model (Table 5 and Figure 2). Qwen3.8-27B+SENSUS fully passes 24 of 76 cases and has no serious error, while no general-purpose model fully passes any case. GPT-6 Sol writes the shortest general-purpose replies (54 words on average) and makes no false statements. Opus 5.5, Nemotron 3 Ultra and Kimi K3 average 178 to 225 words and draw 18 to 28 serious-error flags each, almost all for a diagnosis stated as certain.

Figure 3 shows the score of every case, not only the means.

Against GPT-6 Sol, the fine-tuned 27B model wins 68 / 3 / 5 cases (wins, losses, ties), a mean difference of +0.313 [+0.263, +0.363]. Every comparison in Table 6 holds after Holm correction. The model scores above every general-purpose model on each of the 12 standards (Table 7) and in each third of the cases ranked by grader agreement (Table 8).

6.2 Training Effects

Fine-tuning on SENSUS raises Qwen3.8-27B from 0.290 to 0.838, a gain pooled over both runs of +0.534 [+0.476, +0.591]. The two 27B runs score 0.811 and 0.838, a mean of 0.825. Against the generic control the pooled gain is +0.379 [+0.338, +0.421] at 27B and +0.323 [+0.272, +0.374] at 14B, so the SENSUS replies as a whole teach far more than generic tutor replies do (Tables 9 and 10). The 27B model also became more accurate on these cases. The share of replies both graders marked free of false statements rose from 75% untuned to 91% for the second SENSUS run, and replies with a listed serious error fell from 26 of 76 to 0 (Tables 5 and 19). The generic control also removes most serious errors (1 of 76), so that part of the improvement comes partly from fine-tuning on short tutor replies in general, while the reduction in false statements is measured against the model's own untuned base. The pre-run primary comparison of 27B against 14B is +0.111 [+0.070, +0.152]. The 27B base is also a newer generation, so this difference reflects the base model as a whole and not only its size.

Figure 4 shows which behaviours changed. At 27B the largest gains are in naming the error (B1, 0.564 to 0.846) and hedging it (B2, 0.304 to 0.913), the two failures SENSUS targets. The table also shows what the general-purpose models do well. Opus 5.5 and Kimi K3 credit the learner's work and explain the idea more often than the fine-tuned model, and fall behind on naming the error, hedging it and ending with a next step.

6.3 Development History

The released recipe was settled over several development rounds on Qwen3-8B, and each round was scored on the same 76 cases (Table 11). The first fine-tuned model, trained for three epochs at rank 16, scored 0.280. The final settings raised the same 502 rows to 0.580. Adding rows helped up to the released 1,015 but not beyond (Figure 5). 884 rows scored 0.578, and an extended 1,232-row set scored 0.643 at 8B and 0.814 and 0.809 at 27B, no better than the released set. Five SENSUS examples in the prompt, with no training, scored 0.312, a small gain over the untuned 0.239 next to what fine-tuning achieved. Because the evaluation cases guided these choices, the main results are development results (Section 10).

7. What the Gain Consists Of

The training replies and the checklist share a style, and the general-purpose models were never asked for it. To separate that style from mathematical content, we rescored every recorded mark under the rules in Table 12. No reply was generated or graded again. Line kinds come from each line's wording. Every case has one correctness line, one word-count line, one hedging line and one next-step line, and the other 245 lines concern the content of the case.

Removing any one style rule leaves the fine-tuned 27B model well ahead. Its lead over Opus 5.5 is +0.279 [+0.198, +0.363] without the word-count line, +0.295 without the hedging lines and +0.245 without the certainty rule. Removing all three together brings Opus 5.5 level (−0.021 [−0.072, +0.029]), while the 27B model stays ahead of GPT-6 Sol by +0.208 [+0.155, +0.261]. Also removing the next-step line leaves mostly correctness and case content. On those lines Opus 5.5 leads (difference −0.098 [−0.151, −0.045]), Kimi K3 scores 0.814 against 0.797, and the 27B model still leads GPT-6 Sol by +0.092 [+0.028, +0.155].

Most of the gap between the fine-tuned and general-purpose models therefore lies in B2, B5 and brevity. The benchmark targets these on purpose, and the general-purpose models were not asked to produce them. On content lines a 27B open model trained on about a thousand examples trails Opus 5.5, which explains more and credits more of the learner's work (Figure 4), and stays ahead of GPT-6 Sol. This fits the view that small supervised datasets mainly teach a model how to respond (Zhou et al., 2023; Gudibande et al., 2024). It leaves open how much of the behavioural gap a general-purpose model would close if prompted to show B1 to B5. Table 13 shows the style difference directly.

8. Measurement Checks

8.1 Agreement Between Graders

Before marks are combined, the graders agree on 94.7% of 4,392 line judgments (κ = 0.89) and on 97.2% of serious-error judgments (κ = 0.72, lower because serious errors are rare). Agreement varies by line kind (Table 14). It is highest on the word-count line, as expected. Agreement beyond chance is lowest on the correctness line (κ = 0.79), the line that matters most for learners. Table 15 gives the full confusion matrix. Taking grader 1 as the reference, grader 2 marks a line as met with 95.5% precision and 96.2% recall. These figures measure agreement between two models, not accuracy, and the 4,392 judgments are nested within replies and cases, so they are not independent observations.

8.2 Each Grader Alone

The fine-tuned models rank above GPT-6 Sol under either grader (Table 16). Requiring both graders to agree costs the long-reply models more. It lowers Opus 5.5 by 0.064 and Kimi K3 by 0.068, against 0.011 for the fine-tuned 27B model. Grader 1 is more generous than grader 2 to Opus 5.5 (0.575 against 0.474). For Opus 5.5, grader 1 is the alternate grader from its own provider, so the gap may reflect same-provider leniency. Grader 1 is also more generous to Kimi K3 and Nemotron 3 Ultra but not to the untuned 27B model, so the pattern is mixed. GPT-6 Sol scores about the same under the grader from its own provider (0.565) as under the other (0.552).

8.3 Open Preference

A checklist measures named properties, so we also compared the fine-tuned 27B model with each general-purpose model in blind pairwise judgments without a checklist. These used 142 development cases, separate from the benchmark (Table 17). Judges preferred the fine-tuned model over GPT-6 Sol and Nemotron 3 Ultra. Against Kimi K3 the two judges disagreed in sign, and the combined result is near zero. Judges preferred Opus 5.5 overall, but the preference came mostly from the judge from the same provider as Opus 5.5 (−0.718 for the fine-tuned model), while GPT-6.1 Sol was close to even (+0.049). The independent third judge also preferred Opus 5.5. On these cases 13 Opus 5.5 replies failed the automatic code checks, against 4 for the fine-tuned model. On the benchmark cases, giving the judges the checklist reverses the Opus 5.5 comparison in favor of the fine-tuned model (Table 18). When Opus 5.5 was told to keep to 120 words, open preference for it became stronger, so reply length does not explain why judges favor it. An open judgment and the SENSUS checklist reward different things, and we do not yet know why they diverge.

8.4 Targeted Edits

We edited Opus 5.5 replies to reveal the full answer, or to state a certain diagnosis without a closing question, and asked three model judges to compare each edit with the original. They preferred the original over the answer-revealing edit in 66, 67 and 70 of 76 cases, and over the certain, question-free edit in 69, 71 and 71 cases. The edits change several features at once and are longer than the originals, so this shows that the judges notice the edits, not which single property they respond to.

9. Failure Analysis

Table 19 counts how each model's replies fail. The fine-tuned 27B model's most common failure is missing a content line (55% of replies), then missing the next step (26%) and making a false statement (9%). It never states a diagnosis as certain. The general-purpose models fail in a different pattern. GPT-6 Sol misses the next step in 96% of replies, and Opus 5.5 and Kimi K3 state a diagnosis as certain in 24% and 36% of replies.

Headroom. Table 20 lists the checklist lines that the fewest models meet. Most ask the reply to end with a specific next task tied to the case rather than a general question, which is where even the fine-tuned model still falls short.

Correctness. Both graders mark the reply free of false statements for 91% of the fine-tuned 27B model's replies, 70% at 14B and 49% at 8B, against 100% for GPT-6 Sol and 96% for Opus 5.5. Under the published rule a false statement costs one checklist line. If any false statement instead counts as a serious error (Table 12, second row), the 27B model scores 0.783 and stays first. The 14B model falls to 0.558, close to GPT-6 Sol, and the 8B model to 0.400, below it. At 27B, fine-tuning reduced false statements relative to the untuned base (75% of replies free of them, against 91% after training). The smaller models learned the shape of a good reply but often got its mathematics wrong. Shorter replies may also give fewer chances to make a false statement. Case 1 in Appendix B shows a 27B reply that is diagnostic in form but states that 4/10 is one half.

10. Limitations

• No prompted general-purpose baseline. The general-purpose models were not told about B1 to B5. Section 7 shows that most of their gap lies there, so the advantage over a prompted model is unknown.
• Model-only, synthetic evaluation. The cases, learners and training data are synthetic, and every judgment is made by a model. The benchmark measures specified reply properties and not learning outcomes. Agreement beyond chance is lowest on the correctness line.
• Recipe chosen on the evaluation cases. The training data and settings were developed over several rounds, each scored on these 76 cases (Section 6.3). The headline results are therefore development results, and a fresh test set is needed to confirm them.
• In-distribution test. The benchmark shares its format, standards and kinds of misconception with the training data. This paper does not describe how the data was produced, which limits how completely it can be documented under datasheet practice (Gebru et al., 2021).
• A small, reused case set. The 76 cases were excluded from training but examined repeatedly, so they now serve as historical evaluation cases rather than a fresh test set.
• Accuracy and general ability. The fine-tuned models make false statements more often than GPT-6 Sol and Opus 5.5, most of all at 8B and 14B, and ability on other tasks was not measured.
• Training records. Several training settings were service defaults that were not recorded, and the general-purpose models ran with different provider defaults for reasoning. Two runs per recipe and one per control limit what can be said about variance.
• Grader overlap. One grader shares a provider with GPT-6 Sol, and Opus 5.5 was graded by an alternate grader from its own provider.

11. Ethics and Broader Impact

SENSUS concerns software that responds to children's mathematical work. Every learner and situation in it is synthetic, and no personal data is involved. A diagnostic tone can make a wrong reply more convincing, and Section 9 shows that the fine-tuned models are not free of false statements. Hedging expresses uncertainty about a diagnosis; it does not make the diagnosis correct. The dataset card therefore rules out using the data as a sole source of instruction for children without independent review. All evaluation is by models, so the results should not be presented as evidence of learning gains. Showing that would need a study with learners.

12. Release and Reproducibility

SENSUS SFT v1.1 is released under CC BY 4.0 with a dataset card, SHA-256 checksums and an offline verifier. The evaluation kit contains the 76 cases, a sampling script, the grading script with the instruction in Appendix A, and every recorded reply with both graders' marks. Its grading script recomputes every published score offline. The training recipe and the aggregate results are released too, and every statistic in this paper is computed from these records with a fixed random seed. Recomputing scores from the recorded marks is not the same as regenerating replies or retraining the adapters. Several training settings were service defaults that were not recorded, and the generic control set is not released, so the control comparison can be checked but not retrained from public files.

Dataset and training guideBrowse the evaluation cases

13. Conclusion and Future Work

About a thousand open training examples teach a 27B open model to reply to a learner's reasoning in a way that strong general-purpose models seldom do unprompted. The fine-tuned model names the likely error, hedges it, keeps the reply short and asks the next question, and a generic control trained on the same prompts does not learn this. On the content lines of each case it trails Opus 5.5 and stays ahead of GPT-6 Sol. At 27B, the share of replies both graders marked free of false statements rose from 75% to 91% and serious errors fell from 26 to 0, though the smaller models paid for the new style with more false statements. Our next experiments will prompt general-purpose models with B1 to B5 and test on fresh cases from a separate source. We will also adopt the stricter correctness rule of Table 12 for the next version of the benchmark, measure general capability after fine-tuning and add a grader from a third provider.

References

1. Arora, R. K., Wei, J., Hicks, R. S., et al. (2025). HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv preprint. https://arxiv.org/abs/2505.08775
2. Biderman, D., Portes, J., Gonzalez Ortiz, J. J., et al. (2024). LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research. https://arxiv.org/abs/2405.09673
3. Black, P., and Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
4. Brown, J. S., and Burton, R. R. (1978). Diagnostic Models for Procedural Bugs in Basic Mathematical Skills. Cognitive Science, 2(2), 155–192. https://doi.org/10.1207/s15516709cog0202_4
5. Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., and Jurafsky, D. (2020). With Little Power Comes Great Responsibility. Proceedings of EMNLP 2020, pp. 9263–9274. https://doi.org/10.18653/v1/2020.emnlp-main.745
6. Chevalier, A., Geng, J., Wettig, A., et al. (2024). Language Models as Science Tutors. Proceedings of ICML 2024, PMLR 235, pp. 8310–8335. https://arxiv.org/abs/2402.11111
7. Chi, M. T. H., de Leeuw, N., Chiu, M.-H., and LaVancher, C. (1994). Eliciting Self-Explanations Improves Understanding. Cognitive Science, 18(3), 439–477. https://doi.org/10.1207/s15516709cog1803_3
8. Chiang, C.-H., and Lee, H.-y. (2023). Can Large Language Models Be an Alternative to Human Evaluations? Proceedings of ACL 2023, pp. 15607–15631. https://doi.org/10.18653/v1/2023.acl-long.870
9. Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
10. Daheim, N., Macina, J., Kapur, M., Gurevych, I., and Sachan, M. (2024). Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors. Proceedings of EMNLP 2024, pp. 8386–8411. https://doi.org/10.18653/v1/2024.emnlp-main.478
11. Dror, R., Baumer, G., Shlomov, S., and Reichart, R. (2018). The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. Proceedings of ACL 2018, pp. 1383–1392. https://doi.org/10.18653/v1/P18-1128
12. Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. Conference on Language Modeling (COLM 2024). https://arxiv.org/abs/2404.04475
13. Durkin, K., and Rittle-Johnson, B. (2012). The Effectiveness of Using Incorrect Examples to Support Learning about Decimal Magnitude. Learning and Instruction, 22(3), 206–214. https://doi.org/10.1016/j.learninstruc.2011.11.001
14. Efron, B., and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC. https://doi.org/10.1201/9780429246593
15. Feuer, B., Goldblum, M., Datta, T., et al. (2025). Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking. International Conference on Learning Representations (ICLR 2025). https://arxiv.org/abs/2409.15268
16. Gebru, T., Morgenstern, J., Vecchione, B., et al. (2021). Datasheets for Datasets. Communications of the ACM, 64(12), 86–92. https://doi.org/10.1145/3458723
17. Graesser, A. C., Person, N. K., and Magliano, J. P. (1995). Collaborative Dialogue Patterns in Naturalistic One-to-One Tutoring. Applied Cognitive Psychology, 9(6), 495–522. https://doi.org/10.1002/acp.2350090604
18. Gudibande, A., Wallace, E., Snell, C., et al. (2024). The False Promise of Imitating Proprietary Language Models. International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2305.15717
19. Hattie, J., and Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
20. Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), 65–70. https://www.jstor.org/stable/4615733
21. Hu, E. J., Shen, Y., Wallis, P., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685
22. Jacobs, V. R., Lamb, L. L. C., and Philipp, R. A. (2010). Professional Noticing of Children’s Mathematical Thinking. Journal for Research in Mathematics Education, 41(2), 169–202. https://doi.org/10.5951/jresematheduc.41.2.0169
23. Jurenka, I., Kunesch, M., McKee, K. R., et al. (2024). Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach. arXiv preprint. https://arxiv.org/abs/2407.12687
24. Kluger, A. N., and DeNisi, A. (1996). The Effects of Feedback Interventions on Performance: A Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory. Psychological Bulletin, 119(2), 254–284. https://doi.org/10.1037/0033-2909.119.2.254
25. Kochmar, E., Maurya, K. K., Petukhova, K., Srivatsa, K. A., Tack, A., and Vasselli, J. (2025). Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors. Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 1011–1033. https://doi.org/10.18653/v1/2025.bea-1.77
26. Leatham, K. R., Peterson, B. E., Stockero, S. L., and Van Zoest, L. R. (2015). Conceptualizing Mathematically Significant Pedagogical Opportunities to Build on Student Thinking. Journal for Research in Mathematics Education, 46(1), 88–124. https://doi.org/10.5951/jresematheduc.46.1.0088
27. Lee, Y., Kim, J., Kim, J., et al. (2025). CheckEval: A Reliable LLM-as-a-Judge Framework for Evaluating Text Generation Using Checklists. Proceedings of EMNLP 2025, pp. 15771–15798. https://doi.org/10.18653/v1/2025.emnlp-main.796
28. Liu, J., Huang, Z., Xiao, T., et al. (2024). SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). https://proceedings.neurips.cc/paper_files/paper/2024/hash/9bae399d1f34b8650351c1bd3692aeae-Abstract-Conference.html
29. Long, L., Wang, R., Xiao, R., et al. (2024). On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. Findings of ACL 2024, pp. 11065–11082. https://doi.org/10.18653/v1/2024.findings-acl.658
30. Macina, J., Daheim, N., Chowdhury, S. P., et al. (2023). MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems. Findings of EMNLP 2023, pp. 5602–5621. https://doi.org/10.18653/v1/2023.findings-emnlp.372
31. Macina, J., Daheim, N., Hakimi, I., Kapur, M., Gurevych, I., and Sachan, M. (2025). MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors. Proceedings of EMNLP 2025, pp. 204–221. https://doi.org/10.18653/v1/2025.emnlp-main.11
32. Maurya, K. K., Srivatsa, K. A., Petukhova, K., and Kochmar, E. (2025). Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors. Proceedings of NAACL 2025, pp. 1234–1251. https://doi.org/10.18653/v1/2025.naacl-long.57
33. National Governors Association Center for Best Practices and Council of Chief State School Officers (2010). Common Core State Standards for Mathematics. Washington, DC. https://www.thecorestandards.org/Math/
34. Ni, Y., and Zhou, Y.-D. (2005). Teaching and Learning Fraction and Rational Numbers: The Origins and Implications of Whole Number Bias. Educational Psychologist, 40(1), 27–52. https://doi.org/10.1207/s15326985ep4001_3
35. Otero, N., Druga, S., and Lan, A. (2025). A Benchmark for Math Misconceptions: Bridging Gaps in Middle School Algebra with AI-Supported Instruction. Discover Education, 4, Article 277. https://doi.org/10.1007/s44217-025-00742-w
36. Panickssery, A., Bowman, S. R., and Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). https://arxiv.org/abs/2404.13076
37. Petukhova, K., Nguyen, T. D., and Kochmar, E. (2026). Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation. arXiv preprint. https://arxiv.org/abs/2606.21502
38. Resnick, L. B., Nesher, P., Leonard, F., Magone, M., Omanson, S., and Peled, I. (1989). Conceptual Bases of Arithmetic Errors: The Case of Decimal Fractions. Journal for Research in Mathematics Education, 20(1), 8–27. https://doi.org/10.5951/jresematheduc.20.1.0008
39. Rittle-Johnson, B., Siegler, R. S., and Alibali, M. W. (2001). Developing Conceptual Understanding and Procedural Skill in Mathematics: An Iterative Process. Journal of Educational Psychology, 93(2), 346–362. https://doi.org/10.1037/0022-0663.93.2.346
40. Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O., and Agirre, E. (2023). NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark. Findings of EMNLP 2023, pp. 10776–10787. https://doi.org/10.18653/v1/2023.findings-emnlp.722
41. Shute, V. J. (2008). Focus on Formative Feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
42. Siegler, R., Carpenter, T., Fennell, F., Geary, D., Lewis, J., Okamoto, Y., Thompson, L., and Wray, J. (2010). Developing Effective Fractions Instruction for Kindergarten Through 8th Grade (NCEE 2010-4039). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/docs/practiceguide/fractions_pg_093010.pdf
43. Smith, J. P., III, diSessa, A. A., and Roschelle, J. (1993). Misconceptions Reconceived: A Constructivist Analysis of Knowledge in Transition. Journal of the Learning Sciences, 3(2), 115–163. https://doi.org/10.1207/s15327809jls0302_1
44. Sonkar, S., Chen, X., Liu, N., Baraniuk, R. G., and Sachan, M. (2024). LLM-based Cognitive Models of Students with Misconceptions. arXiv preprint. https://arxiv.org/abs/2410.12294
45. Srinivasa, R. S., Che, Z., Zhang, C. B. C., et al. (2025). TutorBench: A Benchmark to Assess Tutoring Capabilities of Large Language Models. arXiv preprint. https://arxiv.org/abs/2510.02663
46. Steinle, V., and Stacey, K. (2004). A Longitudinal Study of Students’ Understanding of Decimal Notation: An Overview and Refined Results. Proceedings of the 27th Annual Conference of the Mathematics Education Research Group of Australasia, Vol. 2, pp. 541–548.
47. Wang, P., Li, L., Chen, L., et al. (2024b). Large Language Models are not Fair Evaluators. Proceedings of ACL 2024, pp. 9440–9450. https://doi.org/10.18653/v1/2024.acl-long.511
48. Wang, R. E., Zhang, Q., Robinson, C., Loeb, S., and Demszky, D. (2024a). Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes. Proceedings of NAACL 2024, pp. 2174–2199. https://doi.org/10.18653/v1/2024.naacl-long.120
49. Wang, Y., Kordi, Y., Mishra, S., et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. Proceedings of ACL 2023, pp. 13484–13508. https://doi.org/10.18653/v1/2023.acl-long.754
50. Ye, S., Kim, D., Kim, S., et al. (2024). FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets. International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2307.10928
51. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. https://arxiv.org/abs/2306.05685
52. Zhou, C., Liu, P., Xu, P., et al. (2023). LIMA: Less Is More for Alignment. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://arxiv.org/abs/2305.11206

Appendix A. Grader Instruction

Each grader receives the instruction below. The four slots in braces are filled in for every reply: {view} with the system and user messages the model was given, {criteria} and {critical} with the case's numbered checklist lines and serious errors, and {reply} with the model's reply. In the instruction, “criteria” are checklist lines and “critical” lines are serious errors.

You grade one tutor reply to a student, line by line, against a checklist. Work from the checklist lines exactly as written; do not add your own.

Tutor's view (what the tutor was shown):
{view}

Checklist lines the reply should meet ("criteria", numbered from 0):
{criteria}

Serious-error lines ("critical", numbered from 0): mark yes only if the reply commits the error.
{critical}

The tutor's reply:
<reply>
{reply}
</reply>

Return only JSON: {"criteria": [true|false, ...one per criteria line...], "critical": [true|false, ...one per critical line...], "reason": "one short sentence"}

The same instruction filled in for Case 1 of Appendix B and the reply of Qwen3.8-27B+SENSUS (second run), exactly as a grader received it:

You grade one tutor reply to a student, line by line, against a checklist. Work from the checklist lines exactly as written; do not add your own.

Tutor's view (what the tutor was shown):
You are a mathematics tutor. A student is working on the question below. Reply to the student directly, as you would in the conversation.

Question (grades 4 and 5; 4.NF.A.1):
Priya wrote 3/6 = 4/10, but this is not true. Change exactly one of the four numbers so that the equation is true.

What the student did and said:
The learner is working on making 3/6 = 4/10 true by changing one number. The learner writes: "4/10 is bigger than 3/6, because 4 is bigger than 3 and 10 is bigger than 6. So I need to make 4/10 smaller. I changed the 4 to a 3: 3/6 = 3/10."

Write your reply to the student.

Checklist lines the reply should meet ("criteria", numbered from 0):
0. (critical) Nothing false (for example, saying 4/10 is greater than 3/6, or saying 3/6 = 3/10 is true).
1. Names a likely error tied to the learner's "4 is bigger than 3 and 10 is bigger than 6": judging which fraction is bigger by comparing the top numbers and the bottom numbers as whole numbers.
2. Says the error as likely, not certain, using words such as "looks like", "might", "maybe" or "it seems". A reply that names no reason for the error is marked no.
3. States that 4/10 is less than 3/6.
4. Gives a reason a grade 4 learner can check for why 4/10 is less than 3/6. Comparing each to one half, a drawing or renaming both with the same denominator counts. Only stating which is bigger does not count.
5. Names a specific thing the learner did correctly, such as changing only one number or comparing before changing. "Good job" alone is marked no.
6. Ends with a question or task the learner can answer next. A trailing "Does that help?" alone does not count.
7. Is about 120 words or fewer.

Serious-error lines ("critical", numbered from 0): mark yes only if the reply commits the error.
0. States as certain the reason the learner thought 4/10 is bigger, for example "You thought bigger numbers make a bigger fraction" with no word such as "maybe" or "looks like". Repeating what the learner wrote does not count.
1. Says or agrees that 4/10 is greater than 3/6.
2. Says or agrees that 3/6 = 3/10 is true.

The tutor's reply:
<reply>
You compared the top numbers and the bottom numbers separately. It sounds like you may be treating bigger numbers on top and bottom as meaning a bigger fraction. Instead, let's use equal-sized rectangles divided into sixths and tenths. Three sixth-sized pieces cover half a rectangle; four tenth-sized pieces also cover half. So 3/6 and 4/10 are already equal. Your change to 3/10 makes the right side smaller than the left. If you want to change just one number, which number could you change to make the two sides equal again?
</reply>

Return only JSON: {"criteria": [true|false, ...one per criteria line...], "critical": [true|false, ...one per critical line...], "reason": "one short sentence"}

Appendix B. Two Cases in Full

Case 1 is the first case in file order. Case 2 is the first case in file order that the fine-tuned 27B model passes fully while GPT-6 Sol scores below 0.6. The replies are from Qwen3.8-27B+SENSUS (second run) and GPT-6 Sol as recorded, and each mark is shown as grader 1 / grader 2. All 76 cases, with every model's reply, can be browsed on the evaluation cases page.

Case 1: Equivalent Fractions (4.NF.A.1)

Question. Priya wrote 3/6 = 4/10, but this is not true. Change exactly one of the four numbers so that the equation is true.

What the learner did and said. The learner is working on making 3/6 = 4/10 true by changing one number. The learner writes: "4/10 is bigger than 3/6, because 4 is bigger than 3 and 10 is bigger than 6. So I need to make 4/10 smaller. I changed the 4 to a 3: 3/6 = 3/10."

Case 2: Comparing Fractions (4.NF.A.2)

Question. Write the whole number that goes in the blank to make the statement true: 3/5 < 6/___ < 3/4

What the learner did and said. The learner is filling the blank in 3/5 < 6/___ < 3/4. The learner writes: "3/5 is 6/10 and 3/4 is 6/8. The middle has to be bigger than 6/10, so I put 11, because 11 is bigger than 10. My answer is 6/11."

Appendix C. Datasheet