Nature: the reported decline in disruptive science was partly a data artifact

Nature: the debate over the “decline in disruptive science” came down to how bibliographic databases store references

On August 12, Nature published a commentary on a 2023 study that reported a decline in scientific “disruptiveness,” together with a response from the study’s authors. The debate centers on the CD index, a metric that uses later citations to assess whether a paper changed the direction of research.

In the 2023 study, researchers analyzed almost 45 million papers and 3.9 million patents. A work receives a high CD index when later publications cite it without also citing its predecessors. The authors interpreted this pattern as evidence that subsequent work had moved away from the previous line of research.

The problem is apparent in the five-year version of the metric, CD5. If a database record lacks a reference list, a later paper cannot cite both that record and its predecessors. If the record receives at least one later citation, it is assigned the maximum value, CD5 = 1. Missing reference data can therefore raise the mean score by itself.

Vincent Holst and colleagues manually checked random papers and patents in this category. They found references in the original documents for 93% of the papers and 98% of the patents. In SciSciNet, an open database of publications and the citation links between them, the change in mean CD5 from 1944 to 2011 shifted from −0.31 to −0.01 after records with CD5 = 1 were excluded. They argue that the early years appear more “disruptive” because the database did not retain complete reference lists.

In their response in Nature, the authors of the 2023 paper, Michael Park, Erin Leahey, and Russell Funk, attribute the zero-reference records in Holst’s dataset to editorial material, books, conference publications, and reviews. Documents of these types often have no reference list. They also report that alternative metrics designed to exclude documents without references still show a declining trend.

The debate can now be tested through a specific sequence of questions: which documents entered the corpus, where their references disappeared, and how the trend changes after each cleaning step. Before drawing conclusions about the pace of scientific progress, researchers need to determine whether the CD index measures changes in research or properties of the bibliographic database.