A preprint's rule for identifying unusual control groups in mouse lifespan studies flagged the University of Michigan's 2015 male controls: excluding all Michigan males leaves two of three rapamycin regimens without statistical significance
A preprint's rule for identifying unusual control groups in mouse lifespan studies flagged the University of Michigan's 2015 male controls: excluding all Michigan males leaves two of three rapamycin regimens without statistical significance
On October 22, researchers at the University of Texas at San Antonio and The Jackson Laboratory released a preprint (a paper that has yet to undergo peer review). They examined 18 years of data on 10 757 control mice from the US ITP, a program in which three laboratories (these two institutions and the University of Michigan) test substances in parallel to determine whether they extend mouse lifespan.
In the ITP, each substance is tested against control mice that do not receive it, matched by sex, laboratory, and year. If the control mice have unusually short lifespans while the treated mice live as long as usual, the substance appears more effective than it is; if the controls live unusually long, it appears less effective. Median lifespan among male controls ranged from 667 to 927 days across laboratories and years.
Year-to-year variation within laboratories accounts for 71 % of the variation in mean lifespan between male groups (51 % in females): in some years, a laboratory's males live longer on average than is usual for that laboratory, and in others, they live shorter lives. Random differences in group composition account for only 20 %.
The ITP has assessed unusual control groups on a case-by-case basis. In its 2022 report, Texas control mice enrolled in 2017 had unusually short lifespans. The ITP suggested that an unknown environmental factor had shortened lifespan only in the controls and repeated the analysis with Texas excluded. The lifespan increase in males given captopril, a drug for hypertension, was no longer statistically significant, meaning that chance could account for the increase.
The new rule defines a range for each laboratory and sex within which the next control group's mean lifespan is expected to fall with 99 % probability. The range is calculated using only earlier years, with at least eight years of data, and incorporates year-to-year variation, making it wider for males. The authors consider this rule more reliable than searching an existing dataset for unusual groups: in a large dataset, the most extreme group will always look unusual.
The authors tested the rule on historical data, calculating each group's range from preceding years. Of 60 groups, 57 fell within their ranges. Michigan males from 2015 and 2018 and Jackson Laboratory females from 2022 fell outside them. The females lived unusually long, and the 2026 report had already identified them as an outlier with no apparent explanation. The rule did not flag the 2017 Texas males: their short lifespans fell within the usual year-to-year variation for males. In the authors' view, this weakens the basis for treating that group as an outlier and repeating the analysis with Texas excluded.
Michigan males usually live about 10 % longer than males at the other laboratories, but in 2015–2018 they lived substantially shorter lives than usual. ITP researchers had already noted a discrepancy between groups of these males in the 2020 report: males that would later receive rapamycin, which extends mouse lifespan, had lower mortality than controls even before treatment began. Feed from different suppliers was suspected as the cause.
The authors reanalyzed the 2015 rapamycin trial after excluding all Michigan males, both controls and treated mice. Of the three regimens, continuous treatment retained statistical significance (+6,9 % in median lifespan rather than +10,7 %), while the regimen alternating a month of treatment with a month off (−3,6 % rather than +9,3 %) and the three-month course (+6,0 % rather than +10,7 %) lost statistical significance.
The authors propose applying the rule to every control group once follow-up is complete and examining flagged groups through prespecified reanalyses rather than judging them by eye.