AIDaR Workshop Launches a Pilot for Reviewing Scientific Projects in Private GitHub Repositories
AIDaR Workshop Launches a Pilot for Reviewing Scientific Projects in Private GitHub Repositories
On August 29, the organizers of the AIDaR workshop at NeurIPS, a machine learning conference, introduced the voluntary AIDaRS pilot. Manuscripts are still submitted through OpenReview, a platform for submission and peer review. The pilot supplements each manuscript with an anonymized copy of its project folder in a private GitHub repository.
A scientific paper explains its conclusions. In computational research, those conclusions also depend on code, notes about the data, analysis results, and the project’s revision history. AIDaRS gives reviewers access to these files so that they can examine the basis for the conclusions.
After submitting a manuscript, the author provides AIDaRS with the OpenReview link and the project folder. The service creates a temporary copy, removes the previous Git revision history, and scans the files for names and common types of sensitive information. It then places the copy in an anonymized private GitHub repository and opens a pull request, a branch where revisions can be discussed and versions retained. The author reviews the resulting files before submission.
The AIDaRS technical report describes the review process. A reviewer can leave either general feedback or a comment on a specific line in a file. The author receives the comment through the service, modifies the local project folder, and submits either a response or a new version to the same pull request. Each specific comment and the corresponding response remain next to the modified files.
The pilot will test whether GitHub’s review tools produce more useful feedback and make the review process more reproducible. The organizers also want to determine whether a consistently structured project is easier for both people and software agents to read and analyze. Participation is voluntary, while submission through OpenReview remains mandatory.