# Description of the published data

This dataset accompanies the preprint “Benchmarking four large language models on generating and selecting hypotheses that match published findings in 50 studies of ageing biology”: https://eternalsearch.net/omega/benchmark/article.

## Contents

- `generation-outcomes.json`: contains the 2997 hypotheses from the four generators across the final 50 tasks, the primary outcome decided by a majority of the three judges from other companies, the stricter outcome, and the recorded forecasts. The source file and its SHA-256 checksum are also recorded.
- `analysis.json`: contains the generation estimates, intervals, 12 exploratory pairwise comparisons with the Holm correction, extracted forecasting results, and the URLs and checksums of the published reports.
- `review-analyses.json`: contains the additional calculations from Appendix K together with the checksums of the source files.
- `article-selection.json`: contains the saved PubMed query, the download date, the 72 preliminary exclusions, the order of the 212 candidates, the identifiers of the final 50 tasks, and the checksums of the source files. The summary of extraction logs is not a sequential record of all exclusions: entries were updated when replacements were made.
- `judge-scores.json`: contains the scores from 0 to 5 for matching the reference finding from all four judges for each of the 2997 hypotheses, recording the generator, generation method, task, initial hypothesis number, and scores in the order of `judges`. The SHA-256 checksums of all 2400 source scoring files are stored in `fileSha256`.
- `token-audit.json`: contains the sums of `generation_output_tokens` across the 50 tasks for each model and mode, the A/O ratios, and the SHA-256 checksums of all 600 generation files.

The reports from the forecasting experiments are published on the website (Appendix N of the supplementary materials); their URLs and checksums are listed in `analysis.json`. The saved results do not require new calls to the language models.

## Row-level decisions and additional calculations

In `analysis.json`, the `stage` array contains the three generator sets. In each row of `stage[].rows[]`:

- `task` and `id` identify the task and the hypothesis in the published forecasting set;
- `y` equals 1 when the hypothesis matches the reference finding at the primary threshold by a majority of the three judges, and 0 otherwise;
- `forecasts` contains the binary forecasts of the models;
- `selected` equals `true` only if all four models in `selectionDefinition.members` gave a `true` forecast; each of the four models gave a forecast for every hypothesis;
- `coin` contains the previously recorded random binary forecast.

The `selected` field describes the decision of the selection procedure, and the `y` field describes the outcome. The unused `yes` field, which was equal to `false` in the historical reports, has been removed from these rows; the archived source files were not rewritten. The sum of `selected` equals 19, 19, and 22, and the lists have been verified against the three `selected.csv` files.

The identifiers H1-H5 in the forecasting set were assigned after the hypotheses were reordered. To join the judge scores, the pair `task` and `candidate.id` from the corresponding `*-generated-confirmation-20261023/report.json` is used. When joining with `generation-outcomes.json`, one should use the saved `candidate.index` rather than computing the initial index from H1-H5.

`stageThresholds` contains the outcomes of the final selection at thresholds of 3 and 4 for a majority of the judges. `known` denotes the number of all matching hypotheses in the initial set at the same threshold; `matches` denotes the number of matching hypotheses among the selected hypotheses; `selected` denotes the size of the final list. `ppvCi` and `sensCi` were obtained from 10,000 resamples of whole tasks with random seed 20261026. The number `valid` reports in how many resamples the denominator was positive. These are additional retrospective calculations with the lists and the primary threshold held unchanged.

`stageProgress` contains three entries for each generator: the initial set, the hypotheses with a “yes” forecast from Astra and the final list by unanimity of the four models. In each entry, `selected` denotes the number of remaining hypotheses, `matches` denotes the number of hypotheses matching the reference finding among them, and `known` denotes the number of all hypotheses matching the reference finding in the initial set. The metrics `ppv` and `sens` equal `matches/selected` and `matches/known`, respectively; `lostSincePrevious` equals the number of hypotheses matching the reference finding that are present in the previous entry and absent from this one. The intervals `ppvCi` and `sensCi` were obtained from 10,000 joint resamples of whole tasks with random seed 20261026. All entries within a resample use the same tasks. When the denominator is zero, the resample is excluded only from the calculation of the corresponding metric. Before selection begins, sensitivity equals 1 by definition.

`judgeAgreement` contains the agreement of the judges for the binary votes “score of 3 or higher” and “score of 4 or 5”. In `pairs`, for each of the six pairs of judges across all 2997 hypotheses, the entry gives the vote table (`both`, `onlyA`, `onlyB`, `neither`), the proportion of agreeing votes, Cohen’s kappa with its 95% CI, and weighted kappa with quadratic weights on the 0 to 5 scale; the latter matches the value from the original report in `secondary`. In `panel`, for the three judges from other companies that determined the outcome of each hypothesis, the entry gives the distribution of the number of positive votes and Fleiss’s kappa with its 95% CI. The intervals were obtained from 10,000 resamples of the 50 tasks with random seed 20261025. `selectedVotes` contains the same distribution for the hypotheses in the final lists of the three generators.

`sizeControls` contains the calculation for equal-probability selection of a fixed size without replacement. `pool=all` denotes the entire set; a model name denotes only the hypotheses with a positive forecast from that model. `distribution` gives the exact hypergeometric distribution of the number of matches, and `matchesRange95` gives its central 95% range. The `simulation*` fields describe 100,000 random values of the number of matches generated from this distribution to check the mean. Identifiers of random lists were not sampled in this simulation. These ranges reflect the sampling of a list given a fixed set and are not confidence intervals across tasks. The parameters of the additional calculations are stored in `reviewAnalysis`.