OpenBioData separated metadata validation from changes to the final table
OpenBioData separated metadata validation from changes to the final table
On August 5, the OpenBioData developer updated the tool, which searches articles and supplements linked to NCBI records for missing context. It now decides how to handle each field it finds before modifying the metadata table.
An NCBI record may contain a sequence while leaving the participant's age, diagnosis, tissue, or experimental conditions in the article and its supplementary tables. BioSample stores descriptions of biological materials and links them to the associated data and project. Without these fields, researchers cannot determine which samples can reasonably be compared.
The authors of Metappuccino, another tool for reconstructing SRA metadata, demonstrate the extent of the problem. In their sample, 71.6% of the 192 fields returned by the NCBI interface for sequencing run records had missing values. Age and tissue determine the composition of the group used in subsequent secondary analyses.
An OpenBioData user provides a BioProject, BioSample, SRA, GEO, or GenBank identifier, or a link to a publication. The program finds the associated paper, reads the available text and supplements, and extracts the required fields. According to the project description, the output includes the PMID and the name of the specific table or section in which the value was found. An age or diagnosis value therefore retains a direct reference to its primary source: the PMID, table, or section.
The August 5 commit moved validation into the cross_check_fields function. This function decides whether to add a new field, confirm a match, flag a conflict, reject a duplicate identifier, or skip a value. The apply_cross_check function then applies the completed decision to the table. Validation and modification of the table are now separate operations in the code. During secondary analysis, a researcher can trace a populated field back to the passage in the article that supports it.