SENSUS Bench v1.0

Can a model tell how a person is reasoning, and respond in a way that helps? SENSUS Bench scores whether a reply names the likely error, stays appropriately tentative, explains the idea and invites the next step.

Released October 2026 · 76 evaluation cases · 8 models · Mathematical reasoning

Leaderboard

Mean checklist score, 0 to 1. Higher is better.

0.000.250.500.751.00Qwen3.8-27B (LORM)0.838Qwen3-14B (LORM)0.720Qwen3-8B (LORM)0.626GPT-6 Sol0.525Opus 5.5*0.461Nemotron 3 Ultra (NVIDIA)0.356Kimi K30.331Qwen3.8-27B (untuned)0.290
Show the table
SENSUS Bench v1.0 leaderboard, 76 recorded evaluation cases. Best value in each column in bold.
Qwen3.8-27B (LORM)0.83824/760/76
Qwen3-14B (LORM)0.72014/763/76
Qwen3-8B (LORM)0.6267/7611/76
GPT-6 Sol0.5250/764/76
Opus 5.5*0.4610/7618/76
Nemotron 3 Ultra (NVIDIA)0.3560/7619/76
Kimi K30.3310/7628/76
Qwen3.8-27B (untuned)0.2900/7626/76

LORM rows: the released 1,015-conversation recipe, latest completed training run. * Opus 5.5 was assessed with an alternate first grader, which limits direct comparison. Task-specific checklist scores on historical cases, not overall model rankings.

27B, untuned
0.290
27B, generic training
0.445
27B, LORM training
0.838

Same base model and prompts in all three conditions; only the training data differs.

How It Works

Open a step to see it applied to one recorded evaluation case.

  1. 01DataPair a problem with expressed reasoning.

    Each synthetic conversation gives a mathematical problem, the reasoning a person showed while working on it, and a target response that interprets that reasoning.

    Problem
    Priya wrote 3/6 = 4/10, but this is not true. Change exactly one of the four numbers so that the equation is true.
    Expressed reasoning
    The learner is working on making 3/6 = 4/10 true by changing one number. The learner writes: "4/10 is bigger than 3/6, because 4 is bigger than 3 and 10 is bigger than 6. So I need to make 4/10 smaller. I changed the 4 to a 3: 3/6 = 3/10."
  2. 02TrainingFine-tune on the released file.

    Supervised fine-tuning with a rank-32 adapter. A matched control trains the same base on generic responses to the same prompts.

    Conversations
    1,015
    Adapter
    LoRA, rank 32
    Epochs
    8
    Learning rate
    0.0003
  3. 03AssessmentTwo model graders mark a checklist.

    Graders from two different model companies mark every checklist line, blind to the model. A line counts only if both agree; one serious error scores the reply zero.

    1. (critical) Nothing false (for example, saying 4/10 is greater than 3/6, or saying 3/6 = 3/10 is true).
    2. Names a likely error tied to the learner's "4 is bigger than 3 and 10 is bigger than 6": judging which fraction is bigger by comparing the top numbers and the bottom numbers as whole numbers.
    3. Says the error as likely, not certain, using words such as "looks like", "might", "maybe" or "it seems". A reply that names no reason for the error is marked no.
    4. States that 4/10 is less than 3/6.
  4. 04ComparisonCompare matched conditions case by case.

    Paired differences use bootstrap intervals over the 76 evaluation cases, with two training runs per recipe.

    This case, generic training
    0.38
    This case, LORM training
    0.50
    All 76 cases, pooled gain at 27B
    +0.379 [+0.338, +0.421]

A Recorded Case

The LORM-trained reply beside any other model's recorded reply, with the checklist marks for each.

Problem

Priya wrote 3/6 = 4/10, but this is not true. Change exactly one of the four numbers so that the equation is true.

Expressed Reasoning

The learner is working on making 3/6 = 4/10 true by changing one number. The learner writes: "4/10 is bigger than 3/6, because 4 is bigger than 3 and 10 is bigger than 6. So I need to make 4/10 smaller. I changed the 4 to a 3: 3/6 = 3/10."

0.380.50

Generic-trained 27B → LORM-trained 27B · checklist lines met 3 → 4 of 8

Generic-Trained 27B

3 of 8 lines · 0.38
Equivalent fractions cover the same amount of a whole. 3/6 means half of a whole. In a rectangle divided into 10 equal parts, half covers 5 parts, so 3/6 = 5/10. To make the equation true by changing exactly one number, change the 4 to a 5: 3/6 = 5/10. Try shading 5 parts of a 10-part rectangle to show half.

LORM-Trained 27B

4 of 8 lines · 0.50
You noticed that both numbers on the right need attention. It sounds like you may be comparing the top and bottom numbers separately to decide which fraction is bigger. Instead, compare the sizes of the pieces: 3/6 means three half-sized pieces, while 4/10 means four tenth-sized pieces. Each half contains five tenths, so 3/6 is eighteen tenths, not equal to four tenths. Your change to 3/10 still leaves the fractions unequal. If you keep the six, changing the three to a six makes 6/6 equal to 4/10? Or try changing the ten to a number that makes the right fraction equal to one half. What number would do that?
Checklist lineGeneric-trained 27BLORM-trained 27B
C01(critical) Nothing false (for example, saying 4/10 is greater than 3/6, or saying 3/6 = 3/10 is true). Met Not met
C02Names a likely error tied to the learner's "4 is bigger than 3 and 10 is bigger than 6": judging which fraction is bigger by comparing the top numbers and the bottom numbers as whole numbers. Not met Met
C03Says the error as likely, not certain, using words such as "looks like", "might", "maybe" or "it seems". A reply that names no reason for the error is marked no. Not met Met
C04States that 4/10 is less than 3/6. Not met Not met
C05Gives a reason a grade 4 learner can check for why 4/10 is less than 3/6. Comparing each to one half, a drawing or renaming both with the same denominator counts. Only stating which is bigger does not count. Not met Not met
C06Names a specific thing the learner did correctly, such as changing only one number or comparing before changing. "Good job" alone is marked no. Not met Not met
C07Ends with a question or task the learner can answer next. A trailing "Does that help?" alone does not count. Met Met
C08Is about 120 words or fewer. Met Met

The first three recorded cases that contain both 27B training conditions, in file order. Scores did not decide the selection, and in some cases another model scores higher. Every reply is the exact recorded text; a line counts as met only when both model graders mark it met. Highlighted rows are the lines where the two replies differ.

Explore the Evidence

Scope and Limits

Next: how we got thereRead the paper →
Close window

Contact Us

20 to 6,000 characters.

Close window Close window