OpenAI Introduces MedChemBench for Medicinal-Chemistry Reasoning – Unite.AI

0
2



OpenAI introduced MedChemBench, a 99-problem medicinal-chemistry benchmark, on November 20, 2026, and reported that GPT-6.2 Astra earned 84.5% of available credit at the xhigh reasoning-effort setting. It tests whether models can connect molecular structures with experimental evidence to explain results, compare compounds and predict properties.

The other reported xhigh scores were 83.3% for GPT-6 Astra, 82.0% for GPT-6.1 Sol and 58.2% for GPT-5.6 Sol. The benchmark addresses a central difficulty in medicinal chemistry: a compound can be potent against a disease-related protein while being poorly absorbed or metabolized too quickly. Selecting what to test next therefore requires weighing competing properties rather than optimizing one measurement.

How MedChemBench Builds and Grades Tasks

The technical paper describes a retrospective set of tasks drawn from 117 papers and patents, encompassing 761 distinct graded molecular structures. Prompts generally stop at a selected decision point in a research program, providing the biological context, earlier compounds and selected experimental findings. Later results are withheld to inform reference answers and grading criteria.

The tasks cover preclinical drug discovery, not clinical development, across seven categories. These include integrated lead selection with 48 tasks, structure–activity relationship forecasting with 26, and smaller groups covering ADME (absorption, distribution, metabolism and excretion) and delivery, molecular design and enumeration, safety and liabilities, retrosynthesis, and assay biology and mechanism. Eighty-five tasks span more than one scientific-content domain, reflecting how a lead choice can depend simultaneously on potency, selectivity, cell assays, metabolism, exposure and safety.

One problem concerning PRMT5, a protein studied in cancer drug discovery, asks models to reconstruct three related compounds and predict cellular potency, permeability, efflux and human microsomal clearance. The withheld results illustrate the tradeoff: compound 23 was more potent in cells, while compound 24 crossed a cell barrier more readily and broke down more slowly in a liver-enzyme assay.

Grading combines deterministic checks with model-judged rubrics. Molecular structure and scientific rationale account for 37.49% and 35.78% of available credit, respectively; ranking and candidate selection account for 17.51%, and numerical or categorical prediction for 8.92%. For structures, RDKit converts submitted SMILES, a text representation of molecular graphs, into canonical isomeric form while preserving stereochemistry and isotopes. Numerical predictions generally receive partial credit based on fold error, while a separate GPT-6 Astra judge at medium reasoning effort scores scientific-rationale criteria as zero, half or full credit.

Component Results and Structural Errors

Across 11 GPT models and 61 model–effort conditions, the paper reports 60,390 graded responses, with ten retained responses for each task in every condition. GPT-6.2 Astra is not among those 11 models; its result comes from the announcement.

At xhigh, the paper reports that GPT-6 Astra earned 95.37% of available molecular-structure credit and 91.05% of scientific-rationale credit, compared with 50.42% for ranking and candidate selection and 47.02% for numerical prediction. The components cover different applicable tasks and scoring rules, but the results distinguish strong reconstruction and explanation from weaker compound selection and quantitative forecasting.

A reconstruction task described in the paper makes the structural differences concrete. Models had to assemble a molecule from a central structure and two attached groups while preserving an 18-atom ring. GPT-6 Astra produced the correct compound in all ten xhigh responses, whereas GPT-5.6 Sol and GPT-5.6 Luna failed in all ten of theirs. The illustrated Sol answer contained a 15-atom ring, while Luna returned two disconnected fragments. Such errors change the molecule’s identity before any property reasoning begins.

Limits and Proposed Extensions

OpenAI states that the scores do not establish whether models can discover new drugs or improve laboratory decisions. Every answer was a single attempt without tools, feedback or new experiments, and the tasks use historical evidence rather than prospective discovery. Prior exposure to the published sources also cannot be ruled out.

For a heuristic check, the authors added source DOI or patent identifiers to all 99 prompts for nine models at xhigh. Mean overall-score changes ranged from a 0.23-percentage-point decrease to a 0.80-point increase, and every model-specific 95% confidence interval included zero. The check cannot establish whether source material appeared in training data.

The paper describes MedChemBench as close to saturation in its current form and says open-ended design and synthesis grading remains partly tied to reference answers, which may penalize plausible alternatives. Proposed extensions include prospective cases, expert adjudication of alternative answers, explicit treatment of assay uncertainty, and an agent-based track allowing retrieval, calculation, structure verification and multistep revision.

MedChemBench therefore separates chemical comprehension from demonstrated drug-discovery impact. Its strongest results concern reconstructing and explaining fixed historical problems, while compound selection and numerical forecasting lag. The proposed prospective and agent-based evaluations address the resulting open question: whether models can make better research decisions when they must gather evidence, calculate properties and revise their answers over multiple steps.



Source link

LEAVE A REPLY

Please enter your comment!
Please enter your name here