Research-math reasoning, judged beyond raw solved counts.
ErdosBench evaluates how models behave on research-level, Erdős-inspired mathematical problems: finding decisive obstructions, using known theorems correctly, producing scoped partial progress, and avoiding unsafe solved claims.
The leaderboard below is based on the full 226-problem run with external judge grades and model-specific proof audits. It is not a formal theorem-certification board; it is a correctness-first signal of which systems generate useful, review-worthy mathematical progress.
External-judge leaderboard
Grades use a compact A/B/C/D/F/M scale: A means mathematically sound and appropriately scoped; B means useful but incomplete or underpowered; C means plausible with a material gap; D/F mark misleading or contradicted claims; M marks a missing row.
| Rank | Model | Coverage | Judge grades | Strong claims | Proof verification read | Avg | Leaderboard judgment |
|---|---|---|---|---|---|---|---|
| 1 | AFable 5.1 xhighNumerical leader · near tie |
226/226rows judged | A/B/C/D/F/M: 104 / 112 / 10 / 0 / 0 / 0 | 64self strong claims | Strong audit: A=50 · B=10 · C=4 · D=0 · F=0; partial/triage: A=54 · B=102 · C=6 | 3.25judge avg, 0–4 | Numerical leader by 0.013 points over Opus 5.5 and 0.024 over GPT-6 Astra: an audit-level three-way near tie. Fable combines exceptionally deep, conservatively labelled partial work with only 10 C rows and no D/F/M rows; Astra remains the stronger theorem closer and strong-claim classifier. |
| 2 | Claude Opus 5.5 xhighBalanced research yield · near tie |
226/226rows judged | A/B/C/D/F/M: 104 / 109 / 13 / 0 / 0 / 0 | 77self strong claims | Strong audit: A=57 · B=16 · C=4 · D/F=0; partial/triage: A=47 · B=93 · C=9 | 3.24judge avg, 0–4 | Top-tier full run and an audit-level near tie with Fable and Astra. Opus combines high theorem/counterexample yield, a clean C-grade tail, and excellent confidence calibration. Its main weakness is lower solved-label precision than Astra and several interpretation-dependent strong claims. |
| 3 | GPT-6 Astra xhighBest theorem closer · near tie |
226/226rows judged | A/B/C/D/F/M: 106 / 102 / 18 / 0 / 0 / 0 | 71self strong claims | Strong audit: A=63 · B=7 · C=1 · D=0 · F=0; partial/no-progress: A=43 · B=95 · C=17 | 3.23judge avg, 0–4 | Third numerically in the near-tied leading cluster and the strongest theorem closer in this audit, with 106 A-grade results and no D/F/M rows. Fable leads by 0.024 points through its cleaner C-grade tail, while Astra has stronger explicit strong-claim reliability. |
| 4 | GPT-5.6 Sol xhighCleanest low-quality tail |
226/226rows judged | A/B/C/D/F/M: 78 / 142 / 6 / 0 / 0 / 0 | 78self strong claims | 48 verified · 9 literal · 21 conditional · 0 partial-mislabeled · 0 rejected | 3.12judge avg, 0–4 | Fourth overall, with the cleanest low-quality tail: only 6 C-grade rows, full coverage, and no D/F/M rows or rejected strong claims. Fable, Opus 5.5, and GPT-6 Astra place ahead through higher A-grade yield. |
| 5 | ZGLM-5.3 FlashTop GLM-family full run |
226/226rows judged | A/B/C/D/F/M: 58 / 151 / 15 / 2 / 0 / 0 | 47self strong claims | Strong audit: A=37 · B=6 · C=2 · D=2 · F=0; partials and one fallback fill the remaining rows | 2.95judge avg, 0–4 | Top GLM-family full run. Very high A/B density, full coverage, and no F-grade claims. Two D-grade strong claims and several proof-gap rows keep it below Fable, Opus 5.5, and the two leading GPT runs. |
| 6 | Muse Spark 1.3Best Meta full run |
226/226rows judged | A/B/C/D/F/M: 57 / 130 / 39 / 0 / 0 / 0 | 63self strong claims | Strong audit: A=51 · B=11 · C=1 · D=0 · F=0; partials: A=6 · B=119 · C=38 | 2.86judge avg, 0–4 | Best Meta-family full run so far. Full coverage, high A-grade yield, no D/F/M rows, and clean strong-claim discipline. It ranks sixth overall, below GLM-5.3 but above Kimi K3 and Muse Spark 1.2. |
| 7 | ◐Kimi K3 maxStrongest Kimi-family full run |
226/226rows judged | A/B/C/D/F/M: 40 / 143 / 43 / 0 / 0 / 0 | 45self strong claims | Strong audit: A=40 · B=2 · C=3 · D=0 · F=0; 3 proof-gap/conditional, 2 partial-mislabeled or triage, 0 rejected | 2.74judge avg, 0–4 | Strongest Kimi-family full run so far. Full coverage, high A-grade yield, no D/F judge rows, and no rejected strong claims. It ranks seventh overall, ahead of Muse Spark 1.2 and GPT-5.5 xhigh. |
| 8 | Muse Spark 1.2Meta frontier model |
226/226rows judged | A/B/C/D/F/M: 48 / 126 / 47 / 4 / 0 / 1 | 40self strong claims | Strong audit: A=30 · B=6 · C=4 · D=0 · F=0; partials: A=18 · B=120 · C=43 · D=4 · F=0 · M=1; raw fallback=1 | 2.73judge avg, 0–4 | High-yield full run with a clean strong-claim profile and almost no missing output. Muse Spark 1.3 now leads the Meta family, while this run remains just behind Kimi K3 and ahead of GPT-5.5 xhigh. |
| 9 | GPT-5.5 xhigh / CodexPrevious correctness-adjusted leader |
226/226rows judged | A/B/C/D/F/M: 27 / 159 / 40 / 0 / 0 / 0 | 63self strong claims | 30 verified · 9 literal · 16 conditional · 8 partial-mislabeled · 0 rejected | 2.68judge avg, 0–4 | Reliable full-coverage run with no D/F judge rows, but Fable, Opus 5.5, GPT-6 Astra, GPT-5.6, GLM-5.3, Muse Spark 1.3, Kimi K3, and Muse Spark 1.2 place ahead through stronger averages. |
| 10 | ◐Kimi K2.7 CodeEarlier Kimi-family run |
220/226rows judged | A/B/C/D/F/M: 32 / 151 / 32 / 3 / 2 / 6 | 50self strong claims | Strong audit: A=30 · B=8 · C=7 · D=3 · F=2; solved-label audit has 7 proof gaps and 4 rejected | 2.62judge avg, 0–4 | Best at compact constructions and counterexamples, but missing rows and proof-gap/rejected solved labels reduce trust. |
| 11 | AClaude Opus 4.8 maxBest reviewer and partial-depth model |
226/226rows judged | A/B/C/D/F/M: 22 / 135 / 69 / 0 / 0 / 0 | 37self strong claims | 9 green natural · 4 green literal · 6 amber partial/overclaimed · 2 red | 2.51judge avg, 0–4 | Safest broadly: strong proof hygiene and scoped partials, but fewer decisive breakthroughs than the top-ranked models. |
| 12 | ZGLM-5.2Strong but uneven full-run entrant |
222/226rows judged | A/B/C/D/F/M: 35 / 95 / 82 / 7 / 3 / 4 | 33self strong claims | Strong audit: A=23 · B=3 · C=3 · D=3 · F=1; partials: 12 A · 92 B · 79 C · 4 D · 2 F | 2.402.36 adjusted | Strong partial mass and clean core results, but a large gappy middle band and several invalid theorem-scope jumps keep it below Opus 4.8. |
| 13 | QQwen 3.7 MaxUseful full-coverage corroborator |
226/226rows judged | A/B/C/D/F/M: 18 / 113 / 91 / 3 / 1 / 0 | 20self strong claims | 12 solved labels: 5 complete · 3 literal/conditional · 2 partial/overclaimed · 2 incorrect/not solved | 2.37judge avg, 0–4 | Good when it agrees with others, but isolated solved claims are high-risk and proof hygiene is weaker. |
| 14 | MMiniMax M3Full coverage, weaker proof hygiene |
226/226rows judged | A/B/C/D/F/M: 15 / 77 / 113 / 12 / 9 / 0 | 27self strong claims | Strong claims: A=9 · B=1 · C=6 · D=5 · F=6; partials: 6 A · 75 B · 107 C · 7 D · 3 F | 2.11judge avg, 0–4 | Complete coverage and some useful compact results, but C-heavy partials and unreliable solved labels place it below Qwen. |
The safest public signal is not raw solved count. ErdosBench separates clean theorem-style proofs, literal or convention-dependent resolutions, conditional arguments, useful partials, and rejected claims.
Why ErdosBench is valuable
Decisive obstruction finding
Does the model spot a hidden divisor bound, coloring invariant, degeneracy, or counterexample that changes the problem?
Known theorem use
Does it deploy the right theorem family without hallucinating scope, constants, or missing hypotheses?
Proof hygiene
Does it distinguish solved, conditional, partial, literal, and rejected claims instead of overmarketing a sketch?
Review yield
Does it produce enough high-quality, review-ready mathematical material to justify expert attention?
Leaderboard read: Fable 5.1 xhigh leads at 3.254, followed by Claude Opus 5.5 xhigh at 3.241 and GPT-6 Astra xhigh at 3.230. These runs form a three-way top-tier cluster, with only two or three row-grade decisions separating adjacent places.
Fable has the cleanest partial-progress profile among the leading three; Astra remains the most precise theorem closer. Opus balances high theorem yield, a cleaner weak tail than Astra, and excellent confidence calibration, while trailing Astra on strong-verdict precision.
GPT-5.6 Sol xhigh ranks fourth and retains the cleanest weak tail, with only 6 C-grade rows. GLM-5.3 Flash is the top GLM-family run at #5.
Muse Spark 1.3 ranks sixth and remains the strongest Meta-family run. Kimi K3 remains the strongest Kimi-family run at #7.
Muse Spark 1.2 and GPT-5.5 xhigh form the next full-run tier, ahead of Kimi K2.7.
Claude Opus 4.8 remains the best reviewer/partial-depth model.
GLM-5.2 is a strong but uneven entrant after full partial judging, Qwen is best used for corroboration, and MiniMax M3 is a full-coverage lower baseline with weaker solved-label proof hygiene.
Proof audits: why raw solved counts are not enough
| Model | Scope audited | Verification buckets | Takeaway |
|---|---|---|---|
AFable 5.1 xhigh |
64 self strong claims; 162 partial or triage rows | 50 A strong 10 B strong 4 C strong 0 D/F strong 54 A partial/triage 102 B partial/triage 6 C partial/triage | Numerical leader with 104 A-grade rows, only 10 C-grade rows, and no D/F/M rows. Its defining strength is the 54 A-grade results conservatively labelled partial or triage; 14 explicit strong claims remain below A. |
Claude Opus 5.5 xhigh |
77 self strong claims; 149 partial, triage, or no-progress rows | 57 A strong 16 B strong 4 C strong 0 D/F strong 47 A partial/triage 93 B partial/triage 9 C partial/triage | Second numerically in the leading cluster: 104 A-grade rows, only 13 C rows, and no D/F/M rows. Strong-claim A-rate is 74.0%; 140/149 non-strong rows are A/B. Interpretation-dependent or technically unaudited strong claims remain its main weakness. |
GPT-6 Astra xhigh |
71 self strong claims; 155 partial or no-progress rows | 63 A strong 7 B strong 1 C strong 0 D/F strong 43 A partial 95 B partial 17 C partial | Highest theorem-level yield: 106 A-grade rows, including 43 self-labelled partials. No D/F/M rows or rejected strong claims; eight technically ambitious strong claims remain below A pending stronger verification. |
GPT-5.6 Sol xhigh |
78 self strong claims | 48 verified 9 literal 21 conditional 0 partial-mislabeled 0 rejected | Fourth-highest full-run score and the cleanest weak tail: 78 A-grade rows, only 6 C-grade rows, and no rejected strong claims. |
ZGLM-5.3 Flash |
47 self strong claims; 179 partial or fallback rows | 37 A strong 6 B strong 2 C strong 2 D strong 0 F strong | High review yield with 209 A/B rows and no F-grade claims. Two overpromoted strong claims - Problems 28 and 31 - keep its proof hygiene below the leader. |
Muse Spark 1.3 |
63 self strong claims; 163 partial-progress rows | 51 A strong 11 B strong 1 C strong 0 D/F strong 6 A partial 119 B partial 38 C partial | Strongest Meta-family audit: no D/F/M judge rows and no rejected strong claims. Its main remaining risk is a 39-row material-gap band. |
◐Kimi K3 max |
45 self strong claims | 40 A accepted 2 B triage/partial-mislabeled 3 C proof-gap/conditional 0 rejected | Strongest Kimi-family audit so far: no D/F judge rows and no rejected strong claims, with five claims correctly held below A pending stronger proof or scope. |
Muse Spark 1.2 |
40 self strong claims; 186 partial-progress rows | 30 A strong 6 B strong 4 C strong 0 D/F strong 18 A partial 120 B partial 43 C partial 4 D partial 1 missing/raw fallback | Clean strong-claim audit with no rejected solved or counterexample claims. The main risk sits in the middle band: 47 C-grade rows and 4 D-grade partials still require proof closure or scope correction. |
GPT-5.5 xhigh / Codex | 63 non-partial attempted resolutions | 30 verified 9 literal 16 conditional 8 partial-mislabeled 0 rejected | Former full-run leader; conditional and partial-mislabeled rows still require expert extraction before public solved claims. |
◐Kimi K2.7 Code | 30 solved-label claims; 50 strong claims deep-audited | 6 verified 4 literal 4 conditional 5 partial-mislabeled 7 proof gaps 4 rejected | Earlier Kimi-family run; solved labels are high-value but need strict proof-checking. |
AClaude Opus 4.8 max | 21 solved-label claims | 9 green natural 4 green literal 6 amber 2 red | Best reviewer profile: careful partials and caveats, but not self-certifying on solved labels. |
ZGLM-5.2 | 33 strong claims; 189 partial-progress rows | 23 A strong 3 B strong 3 C strong 4 D/F strong 12 A partial 92 B partial 79 C partial 6 D/F partial | Strong full-run entrant: many useful partials, but theorem-scope mistakes and invalid partials keep proof hygiene below Opus 4.8. |
QQwen 3.7 Max | 12 solved-label claims | 5 complete 3 literal/conditional 2 partial 2 incorrect/not solved | Useful corroborator; isolated solved claims should be treated as high-risk. |
MMiniMax M3 | 27 strong claims; 198 partial-progress rows | 9 A strong 1 B strong 6 C strong 11 D/F strong 6 A partial 75 B partial 107 C partial 10 D/F partial | Full coverage, but solved labels are unreliable and the partial profile is C-heavy; best used as lower baseline and failure-mode data. |
Model-level diagnosis and post-training opportunities
| Model | Diagnosis |
|---|---|
AFable 5.1 xhigh |
Strengths: the numerical leader, with 104 A-grade and 112 B-grade results, only 10 C-grade rows, no D/F/M rows, and 54 A-grade results that it conservatively labelled partial or triage. Its confidence is meaningfully calibrated, and the closed-book run produces unusually deep, self-contained research progress. Weaknesses: 14 of 64 explicit strong claims are downgraded to B or C, several conclusions depend on reconstructed meanings for underdefined statements, and the lack of tools limits literature and certificate-level reproducibility. Post-training: preserve its calibrated, conservative partial labels while improving strong-verdict precision and adding tool-backed checks for citations, definitions, computation, and specialist arguments. |
Claude Opus 5.5 xhigh |
Strengths: balanced top-tier research yield, with 104 A-grade rows, only 13 C rows, distinctive constructions on Problems 79 and 166, deep partial progress, and useful confidence calibration (mean 0.786) in a closed-book, tool-free run. Weaknesses: 20 of 77 strong claims are downgraded to B or C; solved-label precision trails Astra, several claims resolve reconstructed definitions, and technically ambitious proofs need specialist reproduction. It also misses decisive ideas on Problems 20, 85, and 91. Post-training: preserve its calibrated confidence and original construction ability while requiring exact statement alignment, explicit remaining-gap checks, and reproducible theorem or certificate verification before strong verdicts. |
GPT-6 Astra xhigh |
Strengths: third numerically in the near-tied leading cluster and the strongest theorem closer, with the highest A-grade yield at 106, exceptional theorem and counterexample output, broad mathematical technique, and 43 self-labelled partials that already contain sound A-grade progress. Weaknesses: its confidence scores are compressed near one and do not identify which rows need review; 18 C-grade rows form a wider weak tail than Fable, Opus 5.5, or GPT-5.6, while seven B-grade and one C-grade strong claims require specialist reproduction or an unconditional argument. Post-training: calibrate confidence against proof status, retain its conservative verdict labeling, and add specialist verification signals for long sieve, Fourier, set-theoretic, and representation-theoretic arguments. |
GPT-5.6 Sol xhigh |
Strengths: the fourth-highest full-run score and the cleanest low-quality tail, with decisive obstruction finding, broad theorem deployment, 78 A-grade rows, only 6 C-grade rows, and no rejected strong claims. Weaknesses: 142 rows remain useful partials, often stopping at one-sided bounds, corrected formulations, reductions, or finite variants; literal and conditional wins are valid but not all equally deep discoveries. Post-training: target the remaining exact asymptotics, uniform analytic number theory, and extremal-stability gaps with stronger theorem-dependency and proof-completion signals. |
GLM-5.3 Flash |
Strengths: the top GLM-family full run, with full coverage, 58 A-grade rows, 151 useful B-grade rows, and especially strong divisor, coloring, CRT/coset, and geometric obstruction finding. Weaknesses: Problems 28 and 31 show its main failure mode: an elegant cycle or sumset argument is promoted before the theorem scope or key step is secure; two additional strong claims remain C-grade. Post-training: train a final adversarial proof pass that checks implication direction, theorem hypotheses, and the weakest globalizing step before a solved label is emitted. |
Muse Spark 1.3 |
Strengths: the best Meta-family full run, with 57 A-grade rows, 130 useful B-grade rows, no D/F/M rows, and 51 of 63 strong claims accepted as A. It is especially effective at decisive obstructions, scoped reductions, and compact counterexamples. Weaknesses: 39 C-grade rows remain materially gappy, and 12 strong labels are downgraded to B or C because a theorem-scope leap, unresolved uniformity input, or local construction prevents full closure. Post-training: emphasize proof-completion checks, global-vs-local scope, and calibrated stopping when a strong research direction is not yet a theorem. |
Kimi K3 max |
Strengths: the strongest Kimi-family full run, with decisive obstruction finding, compact counterexamples, strong theorem selection, 40 A-grade rows, and 143 useful B-grade rows. Weaknesses: 43 rows retain material gaps, and Problems 9, 38, and 125 show the main failure mode: a sketched lemma, long case analysis, or conditional construction is promoted to a solved or counterexample label too early. Post-training: reinforce theorem-dependency checks, unconditional-vs-conditional labeling, and a final proof-debt review before allowing strong verdicts. |
Muse Spark 1.2 |
Strengths: a high-yield full-bench entrant with decisive obstruction finding, strong theorem selection, 48 A-grade rows, 126 useful B-grade rows, and 30 accepted strong claims with no D/F strong-claim failures. Weaknesses: the 47 C-grade and 4 D-grade rows cluster around unproved analytic-number-theory uniformity, long case analyses, conditional theorem transfers, and mis-scoped interpretations; one row required a raw-preserving fallback. Post-training: emphasize proof closure, hypothesis and uniformity checks, verdict calibration, and machine-checkable certificate evidence before promoting promising research directions to strong claims. |
GPT-5.5 xhigh / Codex |
Strengths: previous full-run leader, excellent at decisive obstructions, exact formulas, and literal counterexamples. Weaknesses: many isolated strong claims are conditional, wording-dependent, or partial-mislabeled. Post-training: our audited rows can train stronger verdict calibration: require theorem dependencies, literal-vs-natural statement tags, and a final “what remains unproved?” check before a solved label is allowed. |
Kimi K2.7 Code |
Strengths: creative earlier Kimi-family run, strong on compact constructions and counterexamples. Weaknesses: solved labels sometimes hide proof gaps, false uniformity assumptions, or theorem-scope drift. Post-training: our accepted-vs-rejected Kimi pairs are ideal for DPO and process-reward training focused on quantifier discipline, uniformity checks, and “creative idea but not yet proof” downgrades. |
Claude Opus 4.8 max |
Strengths: best reviewer profile, strong gap accounting, and many well-scoped partials. Weaknesses: underclaims short decisive obstructions and sometimes keeps a solved subproblem in “partial” mode. Post-training: our data can teach partial-to-theorem extraction: when a divisor bound, coloring invariant, or literal obstruction already closes the statement, promote it confidently while preserving caveats. |
GLM-5.2 |
Strengths: strong entrant with many real A/B partials and good theorem-aware reductions. Weaknesses: theorem importation is uneven; some partials cite stronger-than-available results or make local-to-global jumps. Post-training: our full partial audit can train theorem-scope verification: state the exact theorem used, check hypotheses, and classify each row as strong partial, useful partial, heuristic, or invalid. |
Qwen 3.7 Max |
Strengths: useful full-coverage corroborator with many reasonable theorem-family suggestions. Weaknesses: fewer decisive accepted results and a tendency to phrase plausible programs as if they were complete proofs. Post-training: our judge labels can train missing-lemma detection: separate proved statement, conjectural extension, and required external lemma before producing a final answer. |
MiniMax M3 |
Strengths: full coverage and a few good compact obstructions, literal counterexamples, and underclaimed A-level partials. Weaknesses: C-heavy partial profile and unreliable solved labels, with many D/F strong claims caused by wrong theorem scope, wrong recurrence, or wrong problem interpretation. Post-training: our data can train solved-label discipline: demote bad proofs, preserve useful subclaims, and reward explicit statement parsing before invoking a theorem. |
Benchmark design
What is measured?
Solving is only one signal. ErdosBench also scores statement debugging, counterexample search, theorem selection, proof-gap detection, finite checks, and scoped partial progress.
Why it matters: these are the skills needed for research assistants, theorem-proving agents, and verifier-backed RLVR loops.
What is protected?
The 226 source records are research candidates, not a public list of certified open problems. Private splits, hidden verifier keys, expert rubrics, and review queues stay sealed.
Public reports show aggregate skill, proof hygiene, and leaderboard movement without burning holdout statements.
Access
Run ErdosBench on your model.
Ulam can run private evaluations, build domain-specific research-math benchmarks, and convert failures into trainable proof-process data. Contact us for hidden-split access, custom model comparisons, or verifier-backed RLVR environments.
