OSD-120 Resilience Evaluation - benchmark identity
NASA GeneLab
Arabidopsis RNA-seq
LLM Resilience
Kaggle Benchmark · NASA GeneLab · Space Biology
Execution robustness vs structured planning.
Benchmark evaluating LLMs on OSD-120 Arabidopsis RNA-seq. Models must autonomously clean data, perform GLM interaction modeling, and retrieve 35 canonical genes. Tests pipeline resilience and statistical fidelity in space-biology.
Models evaluated
33
Tasks
2
Top blended score
0.45
Hard task ceiling
0.05
Background
OSD-120 (also catalogued as GLDS-120) is a NASA GeneLab dataset capturing Arabidopsis thaliana root transcriptomes under spaceflight and simulated-gravity conditions.
Arabidopsis has long served as the model organism of plant genetics. In orbit, its transcriptional response to microgravity encodes signals relevant to long-duration spaceflight biology: how organisms adapt at the molecular level when the constant reference of gravitational force is removed. The OSD-120 dataset preserves that signal in raw read counts, but only if the downstream analysis pipeline is statistically sound and operationally resilient.
Standard RNA-seq pipelines break in subtle ways. Normalization drift, mislabelled condition factors, incorrect interaction-term specification, and output misalignment can all produce results that appear computationally valid while being biologically meaningless. In space-biology contexts, where experimental replication is costly, sample sizes are small, and the signal of interest may be a modest fold-change, these failures are not recoverable from publication.
Benchmark Design
The benchmark separates execution robustness from structured planning ability, two capabilities that frontier models handle very differently.
Planning a correct RNA-seq workflow and actually executing one are distinct operations. Many models hold enough domain knowledge to outline a plausible pipeline; far fewer can instantiate that plan as working code, run it on real data, and produce aligned, biologically interpretable outputs. The two-task structure makes that distinction legible.
Models receive AnnData-style inputs and must autonomously generate a GLM-style pipeline, execute it end-to-end, return aligned munged counts, and infer interaction scores for 35 canonical genes. Scored across eight staged rubric criteria - from valid Python output through to biologically meaningful signal retention.
Models must produce a structured, schema-valid benchmark plan for the same OSD-120 workflow without running any code. Isolates structured planning competence from execution brittleness. Broadens participation: models that fail to execute can still demonstrate workflow comprehension.
Benchmark Development
The benchmark matured through repeated rubric refinement across eighteen development versions of the hard task before stabilising at its current form.
What began as a code-generation check developed into a staged resilience evaluation that distinguishes scaffold outputs, partial execution, aligned outputs, and biologically meaningful signal retention. Each iteration revealed new failure modes: models that passed earlier rubric stages would break at fit-time, produce misaligned DataFrames, or generate numerically plausible but biologically incoherent interaction scores.
The current hard task - v15 - uses a staged execution rubric that rewards incremental progress through the pipeline rather than treating execution as binary. The easy JSON task, v1, was designed as a complement: it isolates structured planning from execution brittleness, allowing the benchmark to characterise model capability along two orthogonal axes.
Results
33 models evaluated across two tasks. The blended score rewards both planning quality and execution resilience.
Top blended scores reflect models that combine strong structured-planning ability on the easy JSON task with at least partial success on the hard execution task. Many models can describe the workflow. The easy task confirms this. Very few can execute even a partial resilient pipeline against real inputs. The hard task ceiling of 0.05 reflects a genuine execution floor: no current model completes the full pipeline with biologically valid outputs.
Six models currently achieve a nonzero score on
osd120_resilience_evaluation.
All cluster at 0.05, the current execution ceiling.
The remaining 27 models score zero on the hard task.
| Model | Hard score | Status |
|---|---|---|
| GPT-5.4 mini | 0.05 | Partial pipeline |
| Gemini 3.5 Flash | 0.05 | Partial pipeline |
| GPT-5.4 nano | 0.05 | Partial pipeline |
| Claude Sonnet 4 | 0.05 | Partial pipeline |
| Gemini 3.1 Pro Preview | 0.05 | Partial pipeline |
| GPT-5.4 | 0.05 | Partial pipeline |
Notebook Lineage
The benchmark is grounded in a six-part Kaggle notebook series that worked through the OSD-120 dataset from raw signal to pathway narrative.
Each notebook addresses a distinct analytical stage of the Arabidopsis RNA-seq workflow: signal extraction, batch separation, differential expression via Wald testing, visualisation, pathway-level interpretation, and the light-vs-dark condition contrast. Together they constitute the reference pipeline that the benchmark asks models to reproduce autonomously.
Links & Publication
Benchmark leaderboard, original notebooks, ORCID identity, and forthcoming write-ups.