Brian Armstrong proposed sharing data, code and protocols from publicly funded research and paying for independent replication

Brian Armstrong proposed sharing data, code and protocols from publicly funded research and paying for independent replication

On November 7, Coinbase CEO and ResearchHub co-founder Brian Armstrong proposed bringing collaborative open-source software development practices to science. Other teams would have access to the materials for each experiment, repeat it and suggest corrections, with all changes recorded.

Armstrong proposes keeping the materials for each experiment in an open repository with a version history. The data show what was measured, the code shows how those measurements were processed, and the protocol describes the steps performed. The version history identifies the materials used to obtain a result and preserves earlier versions after corrections. Another group could rerun the calculations using the same data or follow the same protocol to conduct an experiment and collect its own measurements. Comparing those measurements with the original data would show whether the group replicated the result in its experiment. If the group found an error in a calculation or the description of the experiment, it could suggest a correction.

The team behind Galaxy, an open platform for biological data analysis, gave thirty AI programs the same paper and received 18 different data analysis plans. In a paper that has yet to undergo peer review, the authors of this test described two omissions in the original paper’s methods that affected the final numbers.

To repeat a laboratory experiment, another team needs people, equipment, supplies and time. Armstrong proposes paying for this work:

Funding organizations could reward replication studies on a par with new papers.

Since 2023, the US National Institutes of Health (NIH) has required researchers it funds to prepare a plan for managing and sharing scientific data. Researchers must comply with the approved plan. Data designated for sharing under that plan should be released as early as possible, no later than the associated publication or the end of the funding period, whichever comes first. This also applies to data that do not lead to a paper.

On November 8, neuroscientist Ruslan Rust supported Armstrong’s proposal:

Ideally, data should be made public as soon as a result is obtained.

Rust writes that he tries to follow this principle in his NIH-funded research. Under his proposed approach, a paper brings together results that other groups have already had an opportunity to examine.

Armstrong argues that AI is increasing the flow of scientific discoveries, so replicating and verifying them will require more work. He identifies the next bottleneck as turning validated discoveries into products, which will require manufacturing and work with regulators.