Users and Processes

The Referee project introduces a new paradigm for evaluatiing academic research, leveraging a universal weakness taxonomy and targeted bug bounties to enhance the quality and reliability of research through a comprehensive bounty system. This ecosystem is built around various user groups that interact with the project according to their needs. This blog post highlights the distinct roles of the project and provides user journeys for Bounty Providers, Bounty Hunters, and Bounty Validators, outlining how each contributes to the overarching goal of advancing academic integrity and innovation

User groups

• General Audience. These people want to check the reliability scores of papers and possibly the connections between papers. They should be able to click on a paper and see how its reliability score was calculated. This group includes educators who guide students through the process of evaluating research reliability and critically assessing research.

• Bounty Hunters (“Breakers”). These people are interested in searching for and collecting bounties. They will have to create and fund an account wallet (submitting a bounty claim requires a small fee to discourage bots). Transactions from these wallets will ensure the order of bounty claim submissions is secure and transparent. This group includes developers who create AI bots or scripts for scanning papers for specific flaws, like Imagetwin and Proofig. These developers may need tools and access to APIs that allow them to test, deploy, and manage their bots within the ecosystem.

• Bounty Providers. These people provide the money for the bounties and thus are the main clients of the project. They also need to create an account, set up a wallet, and fund their account. They should have a dashboard that allows them to create, modify and archive bounties and to track the history of previously set bounties. Bounties will be denominated in either a stablecoin or an existing cryptocurrency store of value (e.g. Bitcoin).

• Bounty Validators. These people review and approve submitted bounties. Bounty providers may assume this role themselves or delegate this responsibility to other registered validators. All validators should be registered, vetted and specialised according to their expertise and interests. This group includes developers who create AI bots or scripts for validating and approving specific bounty claims. These bots will likely proliferate through time as they become more accurate and can be used either to identify potential areas of interest for human reviewers or to sit on jury pools as voting members themselves.

• Paper Authors ("Makers"). These people may provide context for bounty validators, submit their work for review, respond to bounty findings, update their papers based on feedback, and track the reliability scores and critiques of their work. This group may also be interested in staking on their own papers to signify confidence in their research.

• Auditors: These people ensure the overall integrity and fairness of the platform's processes by auditing transactions for anomalies, monitoring for system abuse, and ensuring compliance with platform policies and standards. Auditors may also serve as arbitrators when disputes arise between bounty providers, validators and hunters.

• Research Institutions and Academic Journals: These entities may be interested in monitoring the quality and impact of the research they sponsor or publish. They may also potentially use the platform to pre-screen or post-screen articles for publication or funding.

• Technical Support and User Education: These people contribute by promoting the platform, helping onboard new users, and providing feedback on the platform's usability and policies. They also curate resources or develop educational content about the platform. This group includes Open Science advocates and volunteers.

• Working Group Administrators. These people implement necessary actions as approved by their working groups. See our article on governance for more information on the groups.