Similar Projects

There are numerous open science projects and decentralised science (DeSci) in existence. How does the Referee project differ and why are these differences important? The answer reveals more of our philosophy about where change is most needed. Let’s start by looking at two of the more innovative approaches in the DeSci space.

DeSci Labs/DeSci Foundation were founded by Vrije Universiteit Amsterdam professor Philipp Koellinger. Together, these organisations have an impressive vision for how science research could be improved in the future and their four-part series on the problems with academic publishing is worth reading. Several key innovations include Autonomous Research Communities (ARCs), a Web3-Native Unit of Knowledge, and Secure Persistent Identifiers (PIDs). ARCs are decentralized collectives operating on blockchain technology to curate, validate, and share scientific knowledge securely and transparently, ensuring that the value generated by scientific discoveries is rightfully attributed and rewarded. In addition, these communities can set attestations (constative statements or sets of criteria) that they find valuable, allowing authors to submit a research object to attest to those criteria. Web3-Native Units of Knowledge are intended to replace static PDFs with a dynamic, interoperable format. These facilitate not just the creation and sharing of research but also its verification and reproducibility. Underlying this system are Secure PIDs, which offer a robust alternative to the fragile DOI system; these identifiers are designed to be unbreakable and encode the content of the underlying object rather than merely pointing to its location.

Similar to Desci Labs, the ResearchHub Foundation also seeks to redefine how science is funded, reviewed, and published. Users can earn ResearchCoin (RSC), a community rewards token, for their contributions such as uploading papers, commenting, and posting. They can also receive RSC from other users who appreciate their content or want to tip their papers. ResearchHub also offers an electronic lab notebook for note-takers and has started a pilot for paying peer reviewers, mostly $150 in RSC, for their efforts. Again, this is similar to the Referee project. Like Desci Labs/Foundation, ResearchHub uses the labour theory of value paradigm and doesn’t have a common paper weakness enumeration framework or a reliability scoring system.

Other peer review efforts

• The PubPeer Foundation is a California non-profit that seeks to improve the quality of scientific research by enabling innovative approaches for community interaction. It operates as an open forum where people can post papers and members can comment on them, but there is no formal scoring, reputation staking or downstream processes.

• Review Commons is a journal-independent preprint review platform that follows the traditional model of requesting holistic narrative reviews for papers with the goal of improving their candidacy for publishing.

• The STM Integrity Hub was created by academic journals to provide an environment for publishers to check submitted articles for research integrity issues.

• Ants-Review is a blockchain protocol for incentivizing open and anonymous peer review proposed in 2021 by Bianca Trovò (Sorbonne University) and Nazzareno Massari (MakerDAO). Winner of ETHTurin Hackathon in 2020, this protocol has only been implemented as a proof of concept.

• VitaDAO’s The Longevity Decentralized Review (TLDR) is an on-demand peer review service. Articles from preprint servers are auto-posted daily for review. Reviewers are incentivized to review these papers and receive a share of the donations given to TLDR. Papers and reviews of papers are upvoted by users to quantitatively measure quality. In addition, authors can upvote and comment on reviews to improve feedback and help determine payouts.

• DARPA developed the Systematizing Confidence in Open Research and Evidence (SCORE) program to develop and deploy automated tools to assign "confidence scores" to different social and behavioural science research results and claims. This research relied on surveys and prediction markets to assess the replicability of SBS papers.

• OpenMKT.org aims to increase the transparency of marketing research by tracking direct replications of marketing articles, retractions of marketing articles, preregistered studies with low p-values and studies that evidence of systemic bias in marketing research. There is no formal scoring, reputation staking or downstream processes.

• SCINET is a decentralized research and investment platform focused on the life sciences. Built on the Internet Computer blockchain, it allows retail and institutional investors to invest directly in research and technology with security and authenticity. It is not concerned with evaluating the reliability of existing papers, reputation staking or downstream processes.

• Numerous blogs that document and question papers, such as Data Colada, Research Watch and others.

These projects mostly are led by academics, which tempers their desire to replace the current system radically. As Simine Vazire, professor of psychology at the University of Melbourne and editor-in-chief of Psychological Science, conceded on a Freaknomics podcast, "Our field doesn’t have a culture of open criticism. It’s not considered okay." For this reason, validation is best done by people outside the system as it is in cybersecurity. Referee represents a more radical vision for knowledge curation but is very open to working with members of these projects to advance our mutual objectives.