Hypothesis generation methods in the benchmark
This page describes how the four generator models proposed hypotheses for the benchmark tasks by three methods: the purpose of each method, its sequence of requests, how responses passed between stages, and how the five hypotheses passed to the judges were chosen.
Where generation fits in the study§
GPT-5.6 Sol turned the abstract of each of the 50 articles into a background, a research question and a reference finding with its key elements. The generator received only the background and the question; the judges received the reference finding and decided whether each hypothesis matched it. Predictor models without access to the reference finding then forecast whether each hypothesis would be confirmed experimentally, and the rule of four-model unanimity selected hypotheses for testing first (Section 2 of the preprint, Figure 1).
GPT-6 Astra, Grok 4.5, Claude Opus 5.5 and Gemini 4 Argon generated hypotheses by three methods: a single request, a multistep pipeline and an abridged Omega Point method. Each method produced a set of up to five hypotheses per task. The analysis denominator was 250 hypotheses for each combination of generator and method; 2997 hypotheses were saved, 247 for Claude Opus 5.5 with the Omega Point method and 250 for every other combination. The comparison of the methods is reported in Sections 3.1 and 3.2 of the preprint, and the full text of every prompt is on the prompts page.
Conditions common to the three methods§
Every generation request began with a shared system instruction: the model acts as a research scientist in the biology of ageing, proposes hypotheses that answer research questions and answers in JSON. Stages 1 to 3 asked for free text, whereas the generation stages and the selection request specified JSON; at stage 4 of the Omega Point method the role directive was added to the system instruction. Each hypothesis consisted of a statement, a mechanism, a prediction that would support it over the alternatives, and a short label of its mechanism class. Judges and predictor models received the statement, mechanism and prediction as one text.
The generators ran at high reasoning effort with a limit of 32,000 tokens per request and their own default sampling parameters (Section 2.2). The single request and the request of each stage were repeated when the response could not be parsed or the model refused, with at most three attempts. Refusals by Claude's safety classifier followed the rule of Section 2.3: missing hypotheses counted as non-matching.
Single request§
The single request shows which hypotheses a model proposes directly from the task statement. It is the point of comparison for the multistep pipeline and the Omega Point method, which spend several times as many generated tokens per task (table at the end of the page).
The model received the task and a request to propose five hypotheses in one response (text of the request). The request did not ask for a ranking, so the position of a hypothesis in the response, from first to fifth, is the order in which it was generated. The set consisted of the five hypotheses of the response in the order the model wrote them.
Multistep pipeline§
The multistep pipeline is the control for Omega Point. It keeps the five stages of Omega Point and the transfer of results between them, uses neutral instructions for analysis, generation and improvement of hypotheses, and brings its generated-token output close to that of Omega Point with additional requests. The ratio of the two outputs was 0.978 to 1.119 (Section 3.1 of the preprint), and the multistep pipeline gave the model more candidates to choose from (table at the end of the page).
Each stage was a separate request containing the task and the results of all earlier stages (stage texts):
- analysis of the task: what a good answer must achieve, the main requirements and constraints, and what makes the problem difficult;
- a summary of established knowledge and existing approaches, restricted to what is actually known;
- five to eight of the most important open questions or opportunities, ranked by importance, each with one line of rationale;
- hypotheses that answer the task using the work of the earlier stages;
- an improved version of each of the three strongest hypotheses of stage 4.
The responses of stages 1 to 3 were passed on in full, and the response of stage 4 was turned into a numbered list of hypotheses, each with its statement, mechanism and prediction. After the fifth stage the model received additional requests, each listing every hypothesis already proposed and asking for new, different ones.
Before each additional request its output was estimated from the mean number of generated tokens of all generation calls recorded so far. The request was made if, with this estimate, the total output moved closer to what Omega Point had spent with the same model on the same task. Up to 40 additional attempts were allowed, and the series stopped after the third failed call. When the model refused the Omega Point method on a task, the target was the median output of that model's Omega Point records already saved.
In a final request the model chose from all its candidates the five hypotheses it considered most likely to be confirmed by experiment and most valuable as answers to the research question, best first (text of the request). Valid distinct candidate numbers were kept; when fewer than five remained or the response could not be parsed, the list was filled with the remaining candidates in the order they were proposed.
Omega Point method§
Omega Point was developed at Open Longevity to generate hypotheses with distinct mechanisms. It approaches a task through the mechanisms by which a system fails, looks for contradictions in established findings and for links to distant fields, asks questions about gaps in knowledge, and requires hypotheses that fill assigned creative roles, including one that would force a subfield to revise its core model. The benchmark used an abridged five-stage version without literature search, in which the model drew knowledge of the field from its own training. Each stage was a separate request carrying the results of the earlier stages, passed on as in the multistep pipeline (stage texts):
- inverse failure analysis. The request asked the model to treat the subject of the task as a system and list the ten most common ways it fails catastrophically, above all dynamic failures: feedback loops going wrong, timing failures, cascade collapses and compensatory mechanisms becoming harmful. The failures were to be grouped into three to five clusters named after the failure mechanism, with the healthy steady state each cluster requires;
- frontier map. The request asked for established knowledge bearing on the stage 1 clusters, including at least two contradictions, where two well-supported mechanisms make opposite predictions, and at least two mechanisms imported from unexpected adjacent fields. It also required a dose-response inversion, temporal paradox or context-dependent reversal, evidence that the accepted model may be incomplete, and a measurement approach that would make hidden mechanisms observable;
- gap questions. The request set one of five viewpoints on system failure (Omega Point calls them epistemic lenses) and asked the model to assign each gap between what the clusters require and the frontier map to one of four types: nothing addresses it; the result holds only in isolation; proxies are measured instead of the causal process; findings make opposite predictions. Gaps of the first and fourth types were to be addressed by combining findings from two or more fields. The model then asked five to eight questions of up to 45 words, ranked by consequence, including a mandatory question aimed at falsifying the most promising item. The questions had to include two that invert core assumptions and one each that challenges the accepted framework, imports a quantitative law from another field, and addresses a blind spot between disciplines;
- generation with creative roles. The model answered the task through the questions of stage 3 with hypotheses whose mechanisms differ as much as possible, and the role directive listed below was added to the system instruction. The request also asked the model to consider the physical substrate that stores the state, evolutionary and comparative examples of systems that solved a similar problem, and the way an error signal is broadcast;
- variant evolution. The request asked the model to take the three strongest hypotheses of stage 4 and build one variant of each by one of six operations (refinement, combination, simplification, transfer from another field, inversion of an assumption, switch of target), keeping the parent hypothesis and its prediction. At least one variant was to invert an assumption and at least one to transfer a mechanism from another field.
The role directive of stage 4 required hypotheses of three types in every task. A heretical hypothesis forces a named subfield to revise its core model, rests on a specific puzzling observation, and makes a prediction that would surprise the field. A cross-domain transfer names a law, equation or principle from physics, engineering, ecology or materials science and maps each of its variables to a biological quantity. At least two scout hypotheses propose mechanisms from different non-obvious fields that have empirical grounding and have not yet been applied to the problem. Beyond these roles the directive allowed further hypotheses only for a new mechanism class.
In 10 of the 50 tasks, chosen by a hash of the task identifier, a fourth role was added: a hypothesis that the effect under study is a measurement artefact, an epiphenomenon or several phenomena lumped together. In the other 40 tasks the directive ruled such hypotheses out, because the full Omega Point pipeline grants this role to only a share of the questions of one run.
The five viewpoints of stage 3 describe system failure as loss of agreement between subsystems (distributed consensus), as accumulating errors in storing and transmitting information (information theory and error correction), as loss of robustness of a network of interacting agents (complex adaptive systems), as erosion of safety margins and redundancy (reliability engineering), or as drift of setpoints and weakening of feedback (cybernetic control). The viewpoint was chosen by a hash of the task identifier, so all four generators received the same one on a given task; across the 50 tasks they were used in 12, 10, 11, 11 and 6 tasks, in the order listed.
The model chose five hypotheses from the candidates of stages 4 and 5 with the same final request and rule as in the multistep pipeline. The generated-token output of the Omega Point method was the target for the additional requests of the multistep pipeline.
The full Omega Point pipeline§
The full Omega Point pipeline builds a hierarchy of questions from an initial goal. Its architecture has ten numbered steps:
- formulation of a master question with success criteria;
- decomposition of the master question into goal pillars by inverse failure analysis;
- decomposition of the goal pillars into testable requirements;
- selection of research domains and collection of findings in each of them, including abstracts of recent publications, by two separate model roles;
- matching of the findings to the requirements, performed within the collection of findings at step 4;
- questions about the gaps between requirements and findings;
- hypotheses with creative roles for each question;
- tactical questions that discriminate between competing hypotheses;
- concrete experiments specifying the system, intervention, measurement and time;
- an assessment of whether the experiments of one branch can be combined, with either a common experiment or a justification for separate ones.
Omega Point runs on the site end with the hypotheses of step 7. The abridged benchmark version also ends with hypotheses and starts from a supplied benchmark task instead of a master question. The texts of its first four stages adapt steps 2, 4, 6 and 7 of the full pipeline to a single task, the response format follows the benchmark requirements, and stage 5 uses the variant-evolution operations of Omega Point 2.0. At stage 2 the model gave its account of the field from its own training.
Requests and candidates per set§
The table summarises the saved generation records of the 50 tasks for every combination of generator and method; each cell gives the median with the minimum and maximum across sets. Requests are calls to which the model returned a response, including repeated attempts and the selection request. Candidates are the hypotheses extracted from the response to the single request, from the responses of stages 4 and 5, and from the additional requests; up to five of them entered the set. The generated tokens summed over sets equal the token data of Section 3.1, and the counts for every set are published in a separate file.
| Generator▲ | Method▲ | Requests▲ | Additional requests▲ | Candidates▲ | Stage 4 candidates▲ | Stage 5 candidates▲ | Candidates from additional requests▲ | Generated tokens▲ |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | Single request | 1 (1–1) | 0 | 5 (5–5) | no such stage | no such stage | 0 | 994.5 (725–1455) |
| GPT-6 Astra | Multistep pipeline | 11 (9–15) | 5 (3–9) | 21.5 (14–31) | 4.5 (1–7) | 3 (2–3) | 13 (9–22) | 8250 (6403–11,156) |
| GPT-6 Astra | Omega Point | 6 (6–6) | 0 | 7 (7–9) | 4 (4–6) | 3 (3–3) | 0 | 8299.5 (6581–10,451) |
| Grok 4.5 | Single request | 1 (1–1) | 0 | 5 (5–5) | no such stage | no such stage | 0 | 1433 (881–1903) |
| Grok 4.5 | Multistep pipeline | 9 (8–11) | 3 (2–5) | 41 (31–58) | 6 (5–10) | 3 (3–3) | 31 (20–47) | 16,852.5 (13,791–22,620) |
| Grok 4.5 | Omega Point | 6 (6–6) | 0 | 11 (9–13) | 8 (6–10) | 3 (3–3) | 0 | 15,714.5 (12,787–20,682) |
| Claude Opus 5.5 | Single request | 1 (1–2) | 0 | 5 (5–5) | no such stage | no such stage | 0 | 2718.5 (2101–7631) |
| Claude Opus 5.5 | Multistep pipeline | 9 (6–11) | 3 (0–5) | 30.5 (11–51) | 8 (0–11) | 3 (3–3) | 20 (0–40) | 29,166 (14,430–39,462) |
| Claude Opus 5.5 | Omega Point | 6 (6–6) | 0 | 9 (3–12) | 6 (0–9) | 3 (0–3) | 0 | 27,705 (15,608–35,882) |
| Gemini 4 Argon | Single request | 1 (1–1) | 0 | 5 (5–5) | no such stage | no such stage | 0 | 6988 (2628–15,158) |
| Gemini 4 Argon | Multistep pipeline | 9 (6–13) | 3 (1–7) | 34 (15–68) | 6 (5–8) | 3 (3–3) | 24.5 (6–60) | 101,066.5 (51,940–186,994) |
| Gemini 4 Argon | Omega Point | 6 (6–6) | 0 | 8 (7–15) | 5 (4–6) | 3 (3–9) | 0 | 91,367.5 (59,984–157,088) |