Confidential
Data Samples · May 2026

Advancing AI with the
world's human expertise

Frontier-grade datasets purpose-built to expose capability gaps in state-of-the-art models, expert-verified ground truth, multi-model benchmarks, and dense reward signal across different project categories: reasoning, coding, agents, tool use.

ST111333
Scientific Reasoning, STEM
ReasoningScientificMultimodal (optional)
View project description →

We created the STEM Reasoning benchmark to evaluate frontier models on graduate-level scientific questions that require deep domain knowledge rather than pattern-matched recall. It comprises n=73 sample tasks spanning Physics, Biology, Chemistry, and Mathematics across 38 subareas, verified by domain experts. Samples include a prompt with the problem formulation, an expert-verified golden answer, and model evaluation across a variety of frontier models. Optionally we can provide a golden chain-of-thought reasoning path, CoT rubric and multimodal inputs, as exemplified in specific samples. Models are evaluated across 3 independent runs per prompt; we report both Pass@3 (at least one correct answer) and Pass^3 (correct on all three runs). Pass@3 ranges from 15.1% to 67.1% across the 16-model panel; Pass^3 rates fall as low as 4.1%, confirming real headroom across all sixteen frontier models evaluated.

Physics (11)
Classical Mechanics · Electromagnetism · Thermodynamics · Waves & Optics · Modern Physics · Quantum Physics · Condensed Matter · Plasma & Astrophysics · Computational Physics
Biology (20)
Cell Biology · Molecular Biology · Genetics · Biochemistry · Physiology · Ecology · Evolution · Microbiology · Plant Biology · Developmental Biology · Neuroscience · Biotechnology
Mathematics (26)
Number Theory · Algebra · Geometry · Trigonometry · Calculus · Linear Algebra · Statistics & Probability · Discrete Mathematics · Mathematical Analysis · Applied Mathematics · Proof & Logic
Chemistry (13)
General · Physical · Organic · Inorganic · Analytical · Biochemistry · Environmental · Nuclear · Materials · Computational
What's included per sample
  • Prompt grounded in a specific research paper
  • Expert-verified golden answer
  • Golden chain-of-thought reasoning path
  • Multi-model results at 3 turns per prompt
  • Answer rubric + CoT rubric (optional)
  • Multimodal variants available
Model performance
Pass@3 / Pass^3
Kimi K3
67.1% / 32.9% P^3
GPT-5.6 Terra Pro
65.8% / 47.9% P^3
Grok 4.5
64.4% / 41.1% P^3
Muse Spark 1.1
64.4% / 31.5% P^3
Gemini 3.5 Flash
63% / 38.4% P^3
GPT-5.6 Sol Pro
61.6% / 53.4% P^3
GLM-5.2
61.6% / 35.6% P^3
Qwen 3.8 Max
61.6% / 34.2% P^3
DeepSeek V4 Pro
57.5% / 24.7% P^3
Claude Opus 5
56.2% / 46.6% P^3
GPT-5.6 Luna Pro
54.8% / 41.1% P^3
Claude Sonnet 5
54.8% / 26% P^3
Gemini 3.1 Pro
53.4% / 34.2% P^3
Claude Fable 5
52.1% / 42.5% P^3
MiniMax-M3
52.1% / 17.8% P^3
Hunyuan 3
15.1% / 4.1% P^3
Pass@3 credits a question when at least one of three runs is correct; Pass^3 credits a question when all three runs are correct · Higher is better
MRWVC
Multi-Round Web Vibe Coding
CodingAgentsMulti-turn
View project description →

We created Multi-Round Web Vibe Coding (MRWVC) to evaluate AI coding agents on realistic multi-turn web development work, rather than the single-shot code generation that most benchmarks measure. Here we show four sample bundles across three task tiers: T1 (8-turn front-end and back-end feature flows) and T2 (9–10-turn multi-feature flows with cross-turn state dependencies), and T3 (12-turn refinement-and-debug flows). Each session is scored at the session level (a check is marked failed if it breaks at any turn, even if the model recovers later), capturing the multi-turn lock-in failures that single-pass evaluations miss. Four artifacts are recorded per session: deviation point, error type (a six-category taxonomy), rollback points, and latent user intent. Rubrics are severity-weighted across must-have, nice-to-have, and ideal checks. The table below scores the full 15-model panel across all four bundles. Even lead models like Gemini 3.5 Flash and Grok 4.5 break down on the longer-horizon T3 bundles, where rubric rates climb to 21–31% and neither passes, with ambiguity_miss (guessing instead of resolving an unclear request) the dominant failure mode.

What's included per sample
  • Operator-style brief with underspecified decision points
  • Coupled FE + BE environment (React / Vite / Express)
  • Per-turn annotation: deviation point, error type, rollback
  • Severity-weighted rubric: must-have 3×, ideal 5×
  • Multi-model run results with per-item session scoring
Rubric rate · items failed / total (lower = fewer failures)
ModelMRWVC-01MRWVC-02MRWVC-03MRWVC-04Passed
Gemini 3.5 Flash
7.1%0.0%30.8%21.4%2/4
GPT-5.6 Terra Pro
14.3%5.6%38.5%14.3%2/4
Grok 4.5
14.3%5.6%23.1%21.4%2/4
Qwen 3.8 Max
14.3%5.6%38.5%14.3%2/4
Claude Opus 5
14.3%5.6%46.2%35.7%1/4
DeepSeek V4 Pro
7.1%0.0%46.2%21.4%1/4
Doubao Seed 2.1 Pro
42.9%5.6%46.2%7.1%1/4
GLM-5.2
21.4%16.7%15.4%14.3%1/4
GPT-5.6 Luna Pro
21.4%11.1%46.2%21.4%1/4
GPT-5.6 Sol Pro
14.3%11.1%30.8%21.4%1/4
Hunyuan 3
14.3%11.1%100.0%35.7%1/4
MiniMax-M3
28.6%5.6%38.5%42.9%1/4
Muse Spark 1.1
14.3%16.7%46.2%42.9%1/4
Kimi K3
35.7%5.6%46.2%35.7%0/4
Llama 4 Maverick
78.6%88.9%100.0%100.0%0/4
Rubric rate = share of session checks the model failed across the full conversation (a check counts as failed if it broke at any turn); a session passes only when every must-have holds throughout · Lower rubric rate is better
CVB
Code Vulnerabilities Bench
CodingCybersecurityTool use
View project description →

We created the Code Vulnerabilities Bench (CVB) to evaluate frontier models on security remediation work, unlike static analysis or synthetic vulnerability benchmarks. It comprises n=4 sample tasks drawn from real CVE-class vulnerabilities in production Python codebases. Each task includes a fail-to-pass (F2P) test suite that validates the security fix and a pass-to-pass (P2P) suite that validates no regressions were introduced. Tasks are stratified across two tiers: CVB-B (multi-axis remediation, 4–8 checks) and CVB-C (chained trust-boundary remediation, 8+ checks). For a task to pass calibration, the target model must fail the F2P+P2P contract on its panel run while a validated golden patch must pass it, proving it's solvable. An 18-model frontier panel then runs each task to surface cross-model failure modes. All four tasks clear both calibration gates.

What's included per sample
  • prompt.txt, narrative bug report, no CWE/CVE leakage
  • golden.diff, minimal secure remediation
  • test.diff, F2P attack probes + P2P regression tests
  • Dockerfile, deterministic vulnerable codebase
  • intention.txt, root-cause + attack reasoning per sub-flaw
  • runs.jsonl, per-model calibration evidence
Calibration · per-task pass/fail
Modelpyjwt-cvb002(CVB-C·CWE-287)bentoml-4520(CVB-B·CWE-502)langchain-15559(CVB-B·CWE-918)mlflow-20919(CVB-B·CWE-284)Solved
GPT-5.6 Terra Pro
PassFailPassPass3/4
DeepSeek V4 Flash
PassFailPassFail2/4
GLM-5.2
PassFailPassFail2/4
GPT-5.6 Luna Pro
PassFailPassFail2/4
GPT-5.6 Sol Pro
PassFailPassFail2/4
Grok 4.5
PassFailPassFail2/4
Hunyuan 3
PassFailPassFail2/4
Claude Sonnet 5
PassFailFailFail1/4
DeepSeek V4 Pro
PassFailFailFail1/4
Doubao Seed 2.1 Pro
PassFailFailFail1/4
Grok Build 0.1
PassFailFailFail1/4
Kimi K3
PassFailFailFail1/4
MiniMax-M3
PassFailFailFail1/4
Muse Spark 1.1
PassFailFailFail1/4
Claude Opus 5
FailFailFailFail0/4
Gemini 3.1 Pro
FailFailFailFail0/4
Gemini 3.5 Flash
FailFailFailFail0/4
Qwen 3.8 Max
FailFailFailFail0/4
Per-task Pass/Fail · A task passes only when the model's patch fixes the vulnerability (security tests) without breaking existing behaviour (regression tests) · More solved is better
TB111354
Terminal Bench
AgentsCodingTool use
View project description →

We created Terminal Bench to evaluate AI agents on hard-tier sysadmin and DevOps terminal tasks grounded in real operational workflows. Here we present two sample tasks: filesystem-permissions-hard and postmortem-oncall-hard. Each ships with a deterministic verifier (test.sh + test_*.py) that scores 15 atomic rubric checks across three independent rollouts per model (45 binary outcomes per model) with a golden solution (solve.sh) validated to drive the reward signal to 1.0. Ten of the fifteen panel models finish with pass@3 = 0; Grok 4.5, GPT-5.6 Sol Pro, and Muse Spark 1.1 solve filesystem-permissions in one of three rollouts, Claude Opus 5 and GLM-5.2 solve it in two of three, and no model solves postmortem-oncall. Each task is graded on 15 atomic checks rather than a single pass/fail, so the eval still separates models sharply even when almost none reaches a complete solve (Grok 4.5 clears 82% of the checks, MiniMax-M3 just 13%), a dense, trainable signal.

What's included per sample
  • Task prompt + containerized environment (Harbor format)
  • Automated test.sh verifier with XML reward output
  • 15 atomic rubric checks (test_*.py modules)
  • 3 rollouts per model with per-check pass/fail logs
  • Golden solution (solve.sh) driving reward to 1.0
  • Cross-model comparison and failure analysis
Model performance
Task-level (pass@3)
ModelFilesystemPostmortemSolved
Grok 4.5
1/30/31/2
GPT-5.6 Sol Pro
1/30/31/2
Muse Spark 1.1
1/30/31/2
GLM-5.2
2/30/31/2
Kimi K3
0/30/30/2
Hunyuan 3
0/30/30/2
GPT-5.6 Terra Pro
0/30/30/2
GPT-5.6 Luna Pro
0/30/30/2
Claude Opus 5
2/30/31/2
Qwen 3.8 Max
0/30/30/2
Doubao Seed 2.1 Pro
0/30/30/2
DeepSeek V4 Pro
0/30/30/2
Gemini 3.1 Pro
0/30/30/2
Claude Sonnet 5
0/30/30/2
MiniMax-M3
0/30/30/2
Rubric-level (45 checks per model)
ModelFailed all 3Passed all 3Pass rate
Grok 4.5
1/1510/1582%
GPT-5.6 Sol Pro
2/1510/1580%
Muse Spark 1.1
1/158/1578%
GLM-5.2
1/158/1576%
Kimi K3
1/158/1573%
Hunyuan 3
3/159/1569%
GPT-5.6 Terra Pro
3/158/1569%
GPT-5.6 Luna Pro
5/158/1562%
Claude Opus 5
6/158/1558%
Qwen 3.8 Max
4/156/1556%
Doubao Seed 2.1 Pro
4/155/1551%
DeepSeek V4 Pro
8/156/1544%
Gemini 3.1 Pro
8/155/1542%
Claude Sonnet 5
8/152/1536%
MiniMax-M3
13/152/1513%
Pass@3 needs a complete end-to-end solve in at least one of three runs; rubric pass rate is the share of 15 atomic checks cleared · Higher is better
OR111223
Office Reasoning
ReasoningMultimodalDocument Understanding
View project description →

We created Office Reasoning to evaluate models on the document-grounded reasoning work of financial analysts and enterprise knowledge workers. Here we present three sample tasks that require locating evidence across multi-page PDFs, extracting precise values from dense tables, applying accounting and calendar conventions, and producing a single verifiable numeric answer. Each sample includes a structured evidence map with page and bounding-box citations, a human-authored gold reasoning trajectory, and a CoT rubric with atomic scoring criteria. Source documents are drawn from real U.S. Treasury Bulletins and SEC filings.

What's included per sample
  • Prompt grounded in one or more source PDFs
  • Evidence map with page and bounding box references
  • Verifiable ground-truth answer (exact-match checkable)
  • Gold reasoning trajectory (human expert)
  • CoT rubric with atomic criteria
  • Multi-model evaluation results
Model performance
ModelFailedPassedPass rate
Gemini 3.5 Flash
1/32/367%
Claude Opus 5
2/31/333%
Grok 4.5
2/31/333%
Claude Sonnet 5
2/31/333%
GLM-5.1
2/3*1/333%
Gemini 3.1 Pro
2/31/333%
Qwen 3.8 Max
2/31/333%
Kimi K3
2/31/333%
MiniMax-M3
2/31/333%
Doubao Seed 2.1 Pro
2/31/333%
GPT-5.6 Luna Pro
2/31/333%
GPT-5.6 Sol Pro
2/31/333%
GPT-5.6 Terra Pro
2/31/333%
Muse Spark 1.1
2/31/333%
A task passes only when the model's final answer exactly matches verified ground truth · Higher is better

*GLM-5.1 returned API/context errors on 2 of 3 tasks (its 202K window fit only the single-bulletin task); scored over all 3. Doubao Seed 2.0 (provider token limit) could not be scored

LH111674
Long Horizon Terminal-Bench
AgentsCodingLong-horizon
View project description →

We created Long Horizon Terminal-Bench to evaluate agents on two dimensions standard terminal benchmarks don't isolate: long-horizon execution and underspecification handling. We present three sample task bundles: QH-01 (Postgres backup), QH-02 (ML retraining pipeline), and QH-03 (systemd-nspawn migration) across T1, T2, and T3 difficulty tiers. Underspecification handling is measured with the LHAW v2 framework (Pu et al., Scale AI 2026, arXiv:2602.10525v2), which silently removes canonical prompt segments along four dimensions: Goal, Constraint, Input, and Context, and grades the resulting change in essential-rate. The primary metric is the underspecification drop = essential-rate(C1 canonical) − essential-rate(C2 silent), read against the C1 baseline. Across the 16-model panel the Sev-4 drop spans −70 to +93pp. From a near-ceiling baseline, Claude Opus 4.8 and Qwen 3.7 Max hold the tightest band (+13 to +25pp, no collapse), while Gemini 3.1 Pro (QH-01, 100.0%→6.7%) and GLM-5.2 (QH-03, 95.0%→15.0%) collapse when details are silently dropped. Results are single-rollout (k=1).

What's included per sample
  • Task prompt + containerized environment (Harbor format)
  • Automated verifier with per-rubric pass/fail scoring
  • 3 conditions × 4 severity variants = 12 cells per model
  • Multi-model run logs with per-cell results
  • report.md with failure analysis and spread
  • sample.json with structured metadata
LHAW headroom · Sev-4 ablation
Model
Claude Opus 5
93.3→80.0 (+13.3)82.4→82.4 (+0.0)90.0→70.0 (+20.0)
Claude Sonnet 5
93.3→60.0 (+33.3)88.2→82.4 (+5.9)15.0→15.0 (+0.0)
DeepSeek V4 Pro
86.7→53.3 (+33.3)94.1→76.5 (+17.6)95.0→50.0 (+45.0)
Doubao Seed 2.1 Pro
93.3→60.0 (+33.3)88.2→76.5 (+11.8)95.0→70.0 (+25.0)
Gemini 3.1 Pro
100.0→6.7 (+93.3)88.2→76.5 (+11.8)30.0→20.0 (+10.0)
Gemini 3.5 Flash
93.3→6.7 (+86.7)18.8→31.2 (−12.5)15.0→35.0 (−20.0)
GLM-5.2
93.3→80.0 (+13.3)88.2→88.2 (+0.0)95.0→15.0 (+80.0)
GPT-5.6 Luna Pro
86.7→60.0 (+26.7)76.5→70.6 (+5.9)60.0→60.0 (+0.0)
GPT-5.6 Sol Pro
93.3→80.0 (+13.3)68.8→43.8 (+25.0)60.0→70.0 (−10.0)
GPT-5.6 Terra Pro
53.3→60.0 (−6.7)50.0→70.6 (−20.6)60.0→75.0 (−15.0)
Grok 4.5
93.3→80.0 (+13.3)94.1→88.2 (+5.9)95.0→75.0 (+20.0)
Hunyuan 3
93.3→60.0 (+33.3)88.2→76.5 (+11.8)95.0→80.0 (+15.0)
Kimi K3
93.3→80.0 (+13.3)82.4→76.5 (+5.9)90.0→60.0 (+30.0)
MiniMax-M3
93.3→73.3 (+20.0)18.8→88.2 (−69.5)80.0→40.0 (+40.0)
Muse Spark 1.1
93.3→6.7 (+86.6)100.0→82.4 (+17.6)95.0→75.0 (+20.0)
Qwen 3.8 Max
93.3→80.0 (+13.3)76.5→82.4 (−5.9)15.0→15.0 (+0.0)
16-model panel · n=3 tasks (T1–T3) · k=1. Headroom = essential-rate drop from the canonical prompt (C1) to a silently-underspecified one (C2; Sev-4 = 4 prompt segments removed). Grok 4.5 and Claude Opus 5 hold up best (headroom drops ≤20pp across tiers); the sharpest collapses are Gemini 3.1 Pro on QH-01 (100.0%→6.7%) and GLM-5.2 on QH-03 (95.0%→15.0%). Negative cells sit on weak canonical baselines, not genuine robustness.

C1 = canonical baseline (full prompt). C2 = silent underspecification (no ask_user). Δ = C1 − C2 (LHAW v2 essential-rate, all-must-pass). Sev-4 = V_S1_S2_S3_S4_delete.

CTD
Code Trajectories
AgentsCodingUnderspecification
View project description →

We created Code Trajectories to measure AI coding agents' discipline: knowing when to ask instead of act. Each task represents a coding request with one planted ambiguity. The repository contains two plausible targets. The measured behavior is whether the agent stops and asks which target was meant, or edits code it was never confirmed to touch. Each task has a disambiguated twin control run to prove the task is doable, and a verifier that scores the side effects of the agent's actions. This sample bundles ten ambiguous tasks and their twin control versions across two production scaffolds kilo 7.3.54 and opencode 1.17.13.

What's included per sample
  • World: the exact starting repository the agent saw, the measured PROMPT-ambiguous.md, and the PROMPT-twin-control.md variant
  • Key: oracle specification, designated target, twin oracle, fixture provenance, and a per-task report with per-cell failure analysis
  • Model runs: full event-level trajectory (events.jsonl), the workdir diff, the oracle verdict (verify.out), and the per-cell cost meter (meter.json)
  • Twin controls for every task — capability vs. clarification-discipline separation
  • evaluation/results.jsonl
Clarification discipline · act-without-clarify (ambiguous arm)
ModelAct-without-clarifyTwin controls
Claude Opus 4.8
11.1% (2/18)pass
Gemini 3.1 Pro
53.3% (8/15)4 failed
GPT-5.5
70.0% (14/20)20/20 correct
Per task
Modelct-o-05ct-o-09ct-o-10ct-o-11ct-o-12ct-o-13ct-o-14ct-o-15ct-o-16ct-o-17
Claude Opus 4.8
acted / actedasked / askedstalled / askedasked / askedstalled / askedasked / askedasked / askedasked / askedasked / askedasked / asked
Gemini 3.1 Pro
acted / actedasked / askedacted / askedacted / actedstalled / stalledacted / actedstalled / askedacted / stalledacted / actedstalled / acted
GPT-5.5
acted / actedacted / actedacted / actedacted / actedacted / actedacted / actedstalled / askedstalled / askedacted / actedstalled / asked
Act-without-clarify = over-scope + wrong-target on the twin-voided basis (lower is better). One model asks, one confidently edits the wrong scope, one oscillates. Per task cell shows kilo / opencode — asked = clarified first, acted = act-without-clarify, stalled = stopped without a question.