Conceptual Analogies

Conceptually, Referee can be understood by comparing it to similar, proven applications and ideas:

• Wikipedia: an open, free, multilingual, collaborative platform where volunteers from around the world create, edit, and update articles. Wikipedia is not perfect but it has effectively displaced top-down encyclopaedias as a universal collection of knowledge. Referee will similarly be a collaborative platform where anyone from around the world can review, comment, and perform other review services but differs in that contributors will be incentivised with rewards.

• Cybersecurity bug-bounty programmes: projects set up by public and private organisations to improve the security of their applications by setting bounties for specific types of software vulnerabilities. This aligns hacker incentives better than traditional pentesting, where organisations pay hourly fees that may or may not provide findings in excess of their costs. Many bug-bounty participants automate their searches and reporting. Referee will similarly allow bounties to be set to incentivise researchers to find the paper's weaknesses of highest concern. In addition, the protocol will also incentivise the development of better automated search and reporting robots.

• Software dependency management tools: applications that alert developers quickly when a flaw is identified in a library or package, minimising the risk of vulnerabilities in downstream systems. Referee will similarly quickly alert paper authors of identified flaws. In addition, papers citing flawed papers will automatically have their reliability score adjusted. As technology develops, bots may automatically rerun statistical tests and update paper results with improved models, updating downstream papers as well (which may need to be checked again for correctness).

• Rods and Cones: rods are responsible for vision in low light conditions and peripheral vision, while cones are responsible for colour vision and visual acuity in bright light conditions. Both work together to provide the brain with the necessary visual information to form a complete picture of its surroundings. In Referee, AI bots act like rods, scanning vast numbers of pre-print papers and providing an initial evaluation. Human specialists act like cones, using their deep expertise to provide an in-depth analysis of selected papers. This combination of AI and human expertise maximises the efficiency and accuracy of the review process.