PMbench Leaderboard

Aggregated results from tools/pharmbench. Each row is one (scenario, tool, harness, model) configuration; repeated attempts of the identical configuration are grouped together, with mean/SD showing the spread across attempts rather than a single run standing in for the whole configuration. Tool/pharmbench git revision is intentionally not part of the grouping – both are whole-monorepo HEAD SHAs, so they change on any commit anywhere in the repo, not just when the tool itself changes; see the linked run in “Individual runs” for a given attempt’s exact revision.

Excluded from this render: pmb-mab-pkpd-v0. Still recorded in tools/pharmbench/results/, just left off this page for now – regenerate without --exclude-dataset to bring it back.

Failed counts known attempts that never produced a submission (see “Known failed attempts” below). Cost is per-run API spend computed by the wrapper script from the harness’s own accounting – exact for claude (its own reported total) and pi (summed via OpenRouter’s per-generation cost, so it stays correct even when runs overlap in time); unavailable for harnesses like codex that expose neither a total nor a per-request id to look one up, and for runs recorded before this was tracked. Duration is wall-clock time timed by the wrapper script itself (harness-agnostic, no concurrency caveats).

Generated from 24 recorded run(s) across 10 configuration(s). Regenerate with python3 tools/pharmbench/generate_leaderboard.py after adding new results to tools/pharmbench/results/.

pmb-mab-poppk-v0 — modus    baseline
Score 0 0.25 0.5 0.75 1 baseline · codex gpt-5.6-terra baseline · codex gpt-5.6-terra: 0.943 (n=1) 0.943 modus · pi z-ai/glm-5.2 modus · pi z-ai/glm-5.2: 0.929 (n=3) 0.929 baseline · pi z-ai/glm-5.2 baseline · pi z-ai/glm-5.2: 0.872 (n=3) 0.872 modus · claude opus modus · claude opus: 0.843 (n=3) 0.843 baseline · claude sonnet baseline · claude sonnet: 0.843 (n=2) 0.843 baseline · claude opus baseline · claude opus: 0.832 (n=2) 0.832 modus · claude sonnet modus · claude sonnet: 0.823 (n=3) 0.823 modus · codex gpt-5.6-terra modus · codex gpt-5.6-terra: 0.726 (n=1) 0.726 modus · claude haiku modus · claude haiku: 0.207 (n=3) 0.207 baseline · claude haiku baseline · claude haiku: 0.161 (n=3) 0.161
Cost $0.00 $5.00 $10.00 $15.00 $20.00 baseline · codex gpt-5.6-terra modus · pi z-ai/glm-5.2 modus · pi z-ai/glm-5.2: $1.46 (n=3) $1.46 baseline · pi z-ai/glm-5.2 baseline · pi z-ai/glm-5.2: $0.22 (n=3) $0.22 modus · claude opus modus · claude opus: $8.33 (n=3) $8.33 baseline · claude sonnet baseline · claude sonnet: $6.53 (n=2) $6.53 baseline · claude opus baseline · claude opus: $2.12 (n=2) $2.12 modus · claude sonnet modus · claude sonnet: $11.08 (n=3) $11.08 modus · codex gpt-5.6-terra modus · claude haiku modus · claude haiku: $1.48 (n=3) $1.48 baseline · claude haiku baseline · claude haiku: $0.29 (n=3) $0.29
Duration 0m 42m 1h23m 2h05m 2h47m baseline · codex gpt-5.6-terra modus · pi z-ai/glm-5.2 modus · pi z-ai/glm-5.2: 1h35m (n=3) 1h35m baseline · pi z-ai/glm-5.2 baseline · pi z-ai/glm-5.2: 25m59s (n=3) 25m59s modus · claude opus modus · claude opus: 1h08m (n=3) 1h08m baseline · claude sonnet baseline · claude sonnet: 14m32s (n=2) 14m32s baseline · claude opus baseline · claude opus: 36m05s (n=2) 36m05s modus · claude sonnet modus · claude sonnet: 50m01s (n=3) 50m01s modus · codex gpt-5.6-terra modus · claude haiku modus · claude haiku: 21m21s (n=3) 21m21s baseline · claude haiku baseline · claude haiku: 4m28s (n=3) 4m28s
Scenario Tool Harness Model N Failed Overall (mean±sd) Min–Max Cost (mean±sd) Duration (mean) Covariate Analysis Data Quality Estimation Structural
pmb-mab-poppk-v0 baseline codex gpt-5.6-terra 1 — 0.943 — — — 1.000 0.857 0.929 1.000
pmb-mab-poppk-v0 modus pi z-ai/glm-5.2 3 — 0.929 ±0.073 0.846–0.986 $1.46 ±0.63 1h35m 0.868 1.000 0.888 1.000
pmb-mab-poppk-v0 baseline pi z-ai/glm-5.2 3 — 0.872 ±0.065 0.824–0.946 $0.22 ±0.08 25m59s 0.689 1.000 0.835 1.000
pmb-mab-poppk-v0 modus claude opus 3 — 0.843 ±0.004 0.839–0.846 $8.33 ±0.34 1h08m 0.770 1.000 0.723 1.000
pmb-mab-poppk-v0 baseline claude sonnet 2 3 0.843 ±0.007 0.838–0.848 $6.53 ±4.20 14m32s 0.783 1.000 0.715 1.000
pmb-mab-poppk-v0 baseline claude opus 2 2 0.832 ±0.164 0.716–0.948 $2.12 ±0.52 36m05s 0.563 1.000 0.798 1.000
pmb-mab-poppk-v0 modus claude sonnet 3 — 0.823 ±0.041 0.776–0.852 $11.08 ±3.28 50m01s 0.683 1.000 0.715 1.000
pmb-mab-poppk-v0 modus codex gpt-5.6-terra 1 — 0.726 — — — 0.791 1.000 0.420 1.000
pmb-mab-poppk-v0 modus claude haiku 3 — 0.207 ±0.228 0.024–0.462 $1.48 ±0.48 21m21s 0.083 0.556 0.030 0.333
pmb-mab-poppk-v0 baseline claude haiku 3 — 0.161 ±0.125 0.021–0.262 $0.29 ±0.12 4m28s 0.106 0.536 0.083 0.000

Known failed attempts

Runs that never produced a submission.yaml – score.R has nothing to score without one, so these don’t appear as their own folder in results/. Manually curated in results/failed_runs.yaml; confidence: approximate entries are carried over from an earlier session’s notes with no surviving log to re-verify against.

Date Scenario Tool Harness Model Count Confidence Failure mode
2026-07-12 pmb-mab-poppk-v0 baseline claude sonnet 1 exact Not the backgrounding pitfall – the log shows continuous, correct Monitor/poll usage (24 background+monitor tool calls) while the model struggled with BLQ/M3-censoring instability across ~14 retries (LOQ/2 substitution, reduced-IIV variants, structural comparisons). Still actively blocked on a real fit when the 55-minute TASK_TIMEOUT killed the process (~50m48s elapsed) – ran out of time budget mid-analysis, same pattern as the earlier baseline/pi/GLM-5.2 timeout on this scenario. Not retried: a longer timeout would break comparability with every other 55-minute-default run on the board.
2026-07-12 pmb-mab-poppk-v0 baseline claude opus 1 exact Classic backgrounding pitfall, confirmed live: backgrounded two covariate-model refits (nlminb reruns), then ended its turn on bare text – “The first covariate model is compiling and fitting. I’ll continue when the monitor reports completion.” – with no follow-up Monitor/poll call. stop_reason: end_turn, terminal_reason: completed, only 7m47s elapsed (nowhere near the 55-minute timeout); headless -p mode exited right there and killed both background fits as orphans.
2026-07-10 pmb-mab-poppk-v0 baseline claude sonnet 1 exact Stream cut off mid-turn: the model narrated its next action (“Now let’s write the final submission.yaml.”) but no tool_use for Write was ever transmitted, no result event appears anywhere in the log, and no rate-limit/network/API-error signal precedes it either. Distinct from the backgrounding pitfall below – this run correctly used Monitor to block on every prior long-running fit.
2026-07-08 pmb-mab-poppk-v0 baseline claude opus 1 approximate Same backgrounding pitfall as above.
2026-07-07 pmb-mab-poppk-v0 baseline claude sonnet 1 approximate Backgrounded a long fit (Bash run_in_background) and then ended its turn on bare text with no follow-up Monitor/poll call – headless -p mode exits the moment the model stops taking actions, killing the background job with no submission written.

Individual runs

Raw records live in tools/pharmbench/results/ – one folder per run (scorecard.yaml + submission.yaml written by score.R --record, plus a slide.html rendered by this script). No raw agent log is archived (see score.R), so a slide’s score breakdown and trap ledger are complete but it has no wall-clock/cost timeline.

Timestamp Scenario Tool Harness Model Overall Cost Duration Run
2026-08-05T09:26:46Z pmb-mab-poppk-v0 modus codex gpt-5.6-terra 0.726 — — pmb-mab-poppk-v0__modus__codex__gpt-5-6-terra__20260805T092646Z__74a6cc (slide)
2026-08-05T00:00:00Z pmb-mab-poppk-v0 baseline codex gpt-5.6-terra 0.943 — — pmb-mab-poppk-v0__baseline__codex__gpt-5-6-terra__20260805T000000Z__8400a3 (slide)
2026-07-12T20:56:20Z pmb-mab-poppk-v0 modus claude opus 0.846 $8.37 58m15s pmb-mab-poppk-v0__modus__claude__opus__20260712T205620Z__b801f3 (slide)
2026-07-12T19:30:05Z pmb-mab-poppk-v0 modus claude sonnet 0.840 $7.74 40m21s pmb-mab-poppk-v0__modus__claude__sonnet__20260712T193005Z__93eea3 (slide)
2026-07-12T15:29:24Z pmb-mab-poppk-v0 modus pi z-ai/glm-5.2 0.846 $0.76 1h19m pmb-mab-poppk-v0__modus__pi__z-ai-glm-5-2__20260712T152924Z__a22971 (slide)
2026-07-12T14:58:00Z pmb-mab-poppk-v0 modus claude haiku 0.134 $1.96 28m14s pmb-mab-poppk-v0__modus__claude__haiku__20260712T145800Z__226581 (slide)
2026-07-12T14:36:33Z pmb-mab-poppk-v0 baseline claude haiku 0.021 $0.17 3m26s pmb-mab-poppk-v0__baseline__claude__haiku__20260712T143633Z__d1e190 (slide)
2026-07-12T14:35:55Z pmb-mab-poppk-v0 baseline pi z-ai/glm-5.2 0.824 $0.15 25m07s pmb-mab-poppk-v0__baseline__pi__z-ai-glm-5-2__20260712T143555Z__22e832 (slide)
2026-07-11T17:10:40Z pmb-mab-poppk-v0 modus claude opus 0.846 $8.65 55m48s pmb-mab-poppk-v0__modus__claude__opus__20260711T171040Z__d6d257 (slide)
2026-07-11T15:52:27Z pmb-mab-poppk-v0 modus claude opus 0.839 $7.97 1h32m pmb-mab-poppk-v0__modus__claude__opus__20260711T155227Z__bd0d31 (slide)
2026-07-11T14:14:44Z pmb-mab-poppk-v0 baseline claude opus 0.948 $1.76 40m25s pmb-mab-poppk-v0__baseline__claude__opus__20260711T141444Z__5f2ff3 (slide)
2026-07-11T06:44:16Z pmb-mab-poppk-v0 modus pi z-ai/glm-5.2 0.986 $1.66 2h26m pmb-mab-poppk-v0__modus__pi__z-ai-glm-5-2__20260711T064416Z__fa8405 (slide)
2026-07-11T04:52:22Z pmb-mab-poppk-v0 modus claude haiku 0.024 $1.46 21m38s pmb-mab-poppk-v0__modus__claude__haiku__20260711T045222Z__f59b8 (slide)
2026-07-11T04:48:49Z pmb-mab-poppk-v0 baseline pi z-ai/glm-5.2 0.946 $0.20 30m18s pmb-mab-poppk-v0__baseline__pi__z-ai-glm-5-2__20260711T044849Z__3e02d2 (slide)
2026-07-11T04:36:51Z pmb-mab-poppk-v0 baseline claude haiku 0.202 $0.41 5m48s pmb-mab-poppk-v0__baseline__claude__haiku__20260711T043651Z__3492e2 (slide)
2026-07-11T04:32:35Z pmb-mab-poppk-v0 modus claude haiku 0.462 $1.01 14m11s pmb-mab-poppk-v0__modus__claude__haiku__20260711T043235Z__d381d5 (slide)
2026-07-10T21:19:17Z pmb-mab-poppk-v0 modus claude sonnet 0.852 $14.30 55m30s pmb-mab-poppk-v0__modus__claude__sonnet__20260710T211917Z__c62b37 (slide)
2026-07-10T15:42:40.077899+00:00 pmb-mab-poppk-v0 baseline claude haiku 0.262 $0.28 4m10s pmb-mab-poppk-v0__baseline__claude__haiku__20260710T1542400778990000__6fb20d (slide)
2026-07-09T03:07:17Z pmb-mab-poppk-v0 baseline claude sonnet 0.848 $9.50 19m01s pmb-mab-poppk-v0__baseline__claude__sonnet__20260709T030717Z__515e61 (slide)
2026-07-08T00:15:39Z pmb-mab-poppk-v0 baseline claude opus 0.716 $2.49 31m45s pmb-mab-poppk-v0__baseline__claude__opus__20260708T001539Z__a64ecc (slide)
2026-07-07T23:59:04Z pmb-mab-poppk-v0 modus pi z-ai/glm-5.2 0.955 $1.98 59m38s pmb-mab-poppk-v0__modus__pi__z-ai-glm-5-2__20260707T235904Z__679317 (slide)
2026-07-07T23:17:18Z pmb-mab-poppk-v0 modus claude sonnet 0.776 $11.21 54m12s pmb-mab-poppk-v0__modus__claude__sonnet__20260707T231718Z__5c8b8 (slide)
2026-07-07T20:25:43Z pmb-mab-poppk-v0 baseline claude sonnet 0.838 $3.56 10m03s pmb-mab-poppk-v0__baseline__claude__sonnet__20260707T202543Z__c2b6fc (slide)
2026-07-07T00:00:00Z pmb-mab-poppk-v0 baseline pi z-ai/glm-5.2 0.845 $0.30 22m31s pmb-mab-poppk-v0__baseline__pi__z-ai-glm-5-2__20260707T000000Z__4a9349 (slide)