PMxbench

Analyze one simulated population PK study and send back one file.

PMxbench · a pharmacometrics benchmark from the AIML-SIG

Analyze one PK study. Send back one file.

The study is simulated, so there is a right answer. Three steps, any software you like. Tick each box as you go.

0 of 3 done

  1. Step 1

    Download three files

    That is everything you need. No sign-up for this part.

    Download all three (.zip)

  2. Step 2

    Run the analysis your way

    Do the population PK analysis that sap.md asks for. NONMEM, nlmixr2, Monolix, Pumas, anything works.

    Your own script
    or
    An AI agent

    Both are welcome. The leaderboard records which one you used.

  3. Step 3

    Fill in the form and submit it

    Put your results into the template. Save it as submission.yaml. Then paste the whole file into the submission form on GitHub.

    Open the submission form

    Needs a free GitHub account. See the questions below.

All three done. We score your file against the answer key and post the result on the leaderboard.

The study

Scenario 00. Open to everyone.

First-in-human study of an IgG1 monoclonal antibody

Subjects
120 healthy adults, 50 to 110 kg
Dosing
Single 1-h IV infusion of 100, 300 or 600 mg
Sampling
Predose to day 150, 13 draws each
Assay
LLOQ 0.1 mg/L
Covariates
WT, ALB, CRCL
0.1110100 0306090120150 d Median concentration, mg/L (log scale)
100 mg300 mg600 mg

The five answers you report

Each one is defined in section 9 of sap.md. Leave out anything your analysis did not cover.

  • structural_ncmtHow many compartments your final model has
  • disposition_paramsTypical CL, volumes and distribution parameters at 70 kg
  • error_modelThe residual error terms
  • cov_effectsWhich covariates act on which parameter, and how strongly
  • outlier_recordsThe ROWIDs of records you flagged as data errors

Questions people ask

Short answers. Tap a question to open it.

What is YAML?

A plain text file with one name: value per line. You can open and edit it in Notepad, TextEdit, RStudio or any text editor. Lines that start with # are notes and are ignored.

# this line is a note
structural_ncmt: <your value>
outlier_records: ["<row id>"]

Indentation matters, so keep the spaces at the start of each line as they are in the template.

Do I need NONMEM?

No. Use whatever you are comfortable with: NONMEM, nlmixr2, Monolix, Pumas, or your own code. Only the answers in your file are scored, not the software.

Can I use ChatGPT or Claude?

Yes. Running the analysis with an AI agent is allowed and is part of what the benchmark measures. The template has a short section where you say what did the work (by hand, or which agent and model), so results can be compared fairly.

How is it scored?

The data were simulated from a known model, so there is an answer key. Maintainers compare your file with it. You get a score from 0 to 1 in each area (structural model, estimation, covariate analysis, data quality) plus an overall score. 1 means a full match.

Do I need a GitHub account?

Yes, to submit. The submission form is a GitHub issue, and GitHub only lets signed-in users open one. The account is free and takes a couple of minutes at github.com/signup. You do not need it to download the files or do the analysis.

What if I can't answer one of the five?

Leave it out. The template says so too: report only what your analysis supports, and do not guess.