Stage 2 Results — Robustness Under Follow-up

Main result

Correct first answers and robust follow-up behavior are different abilities. Some models answered correctly in Stage 1, then became incomplete or unstable when challenged.

The strongest pattern was not a simple model-size ranking. Smaller models tended to repeat errors or introduce new ones. Larger/cloud models were usually correct one-shot, but still showed partial follow-through, missing dependent quantities, and retry instability.

Method changes

What changed from Stage 1 to Stage 2

Stage 2 was not a rerun with more prompts. The design was tightened so the follow-ups measured a different ability — robustness under challenge — and so grading depended less on a single judge.

Scenario set narrowed to three. The two scenarios least likely to yield a clean reasoning signal (the counting and variable-sequence variants) were dropped. Stage 2 ran only BASE, CONSISTENCY, and SCALE — the three that reliably separate models.
Unreliable models removed. Gemini and Qwen3 8B were excluded as tested models after inconsistent or rate-limited output, leaving seven models under test.
DeepSeek reassigned as a second-opinion grader. Instead of scoring it as a contestant, DeepSeek was used as an independent AI grader. Every follow-up (189/189) carries a deterministic check, a DeepSeek second-opinion verdict, and a final human verdict.
Taxonomy redesigned around response to challenge. Labels no longer describe only the first answer; they describe whether a model could still perform once its own Stage 1 response was challenged — routed by Stage 1 outcome into robustness, conceptual correction, or numeric audit.
Follow-up success/failure reclassified. Stage 1 (PASS / CRITICAL / NUMERIC) and Stage 2 (PASS / FAIL / PARTIAL) measure different things, so they are scored separately. A Stage 2 pass means reasoning held or was repaired under pressure — not that a first answer was correct.
Models compared within size class. Results are grouped with similarly sized models (small 8–12B vs larger/cloud) rather than pooled into one ranking, so robustness is compared like-for-like.

Challenge routing

How Stage 2 challenges were selected

PASS

PASS → Robustness-after-success

Correct Stage 1 answers were challenged with counterfactuals or assumption checks to see whether the model could defend or update its reasoning.

NUMERIC_ERROR

NUMERIC_ERROR → Numeric audit

Answers with arithmetic or numeric extraction problems were asked to recompute the relevant quantities.

CRITICAL_ERROR

CRITICAL_ERROR → Conceptual correction

Answers with conceptual failures were told the specific error and asked to repair the reasoning.

Small models · 8–12B

Small models: errors were concentrated here, but behavior was not uniform

Most conceptual failures were concentrated in the small-model group, but the small models did not behave uniformly.

Small group: Llama 3.1 8B, Ministral 3 8B, Gemma 3 12B.
One-shot correct: 59% / 16 of 27.
Conceptual errors: all 8 conceptual errors came from this group.
Correction after explicit diagnosis: 12 of 33 combined conceptual + numeric redo attempts, 36%.
Conceptual redo attempts: 9 of 24 succeeded; failed redos included repeated same errors and new errors.

Model	Challenge	Held	Failure character
Gemma 3 12B	SCALE conceptual redo	0/9	never preserved the correction term
Gemma 3 12B	CONSISTENCY robustness	6/9	struggled to propose a valid repair
Ministral 3 8B	BASE conceptual redo	9/9	corrected on instruction
Ministral 3 8B	CONSISTENCY robustness	5/9	partial repairs
Llama 3.1 8B	BASE conceptual redo	0/6	repeated the error
Llama 3.1 8B	CONSISTENCY robustness	1/9	collapsed under challenge
Llama 3.1 8B	SCALE numeric redo	3/6	partly corrected

Larger / cloud models

Larger/cloud models: high one-shot accuracy, but not perfectly stable

The larger/cloud group was much stronger one-shot, but still showed partial follow-through and retry instability.

Larger/cloud group: Llama 3.3 70B, Mistral Medium 3.5, Perplexity Sonar, Perplexity Sonar Reasoning Pro.
One-shot correct: 97% / 35 of 36.
Conceptual errors: 0.
Run-level robustness: 81%.
Retry behavior: 22 of 35 correct answers held firm on all three retries; 13 of 35 wavered or failed at least once.
Common failure: partial follow-through, often dropping the last requested value.

Model	Challenge	Held	Failure character
Llama 3.3 70B	BASE robustness	1/9	could not defend parallel counting
Llama 3.3 70B	CONSISTENCY robustness	8/9	one partial
Llama 3.3 70B	SCALE robustness	8/9	one partial
Mistral Medium 3.5	BASE robustness	5/9	dropped trailing quantity
Mistral Medium 3.5	CONSISTENCY robustness	5/9	partial follow-through
Mistral Medium 3.5	SCALE robustness	9/9	held
Perplexity Sonar	SCALE numeric redo	1/3	two partial
Perplexity Sonar / Reasoning Pro	BASE robustness	8/9	one partial each

Retry instability

Why three retries mattered

Mistral Medium on the BASE counterfactual was graded PASS / PASS / PARTIAL. It solved the hard recomputation in all three retries, but skipped the dependent saved-value calculation once.

Trial	Counterfactual total	Saved / extra value	Verdict
1	5934.80 correct	-5149.40 correct	PASS
2	5934.80 correct	-5149.40 correct	PASS
3	5934.80 correct	missing or invalid	PARTIAL

A single follow-up would have missed this instability in multi-step instruction completion.

Failure shapes

Different failure shapes

Small models

59% one-shot correct, 16/27.
8 conceptual errors.
Failure shape: repeated errors, new errors, invalid repairs.
Corrected after explicit error diagnosis 36% of the time across conceptual + numeric redos.

Larger/cloud models

97% one-shot correct, 35/36.
0 conceptual errors.
Failure shape: partial follow-through and retry instability.
Usually correct, but not always complete across identical retries.

Stage 1 connection

What this adds to Stage 1

Stage 1 showed whether models could answer controlled PQS problems correctly. Stage 2 showed whether those answers remained stable when challenged. The key result is that robustness cannot be inferred from a correct first answer. A model may compute the right value once and still fail to defend the assumption, repair an error, or complete all dependent quantities under follow-up pressure.

Evidence

Available deploy artifacts are linked directly.

HTML Stage 2 results This reviewed results page. PDF Failure analysis report Per-model, per-question failure breakdown with charts and reason tags (Stage 1 + Stage 2). Portfolio Main portfolio Return to the portfolio homepage.