Referee
  • Introduction
    • Academic Peer Review Today
    • Similar Projects
    • How Referee is Different
    • Limits of Current Practice
    • Needed Improvements
    • Proposed Solution
    • Conceptual Analogies
    • Stakeholders
    • Impact
  • System Design
    • Technical Architecture
    • Governance
      • Working Group (WG): CRWE
      • WG: Reliability Scoring Method
      • WG: AI Agents
      • WG: Bounty Setting & Claiming
      • WG: Marketing and Outreach
      • WG: Technical Architecture
      • WG: Specialised Academic Guilds
    • Users and Processes
      • User Journey: Bounty Provider
      • User Journey: Bounty Hunters
      • User Journey: Bounty Validators
    • Common Research Weakness Enumeration (CRWE)
Powered by GitBook
On this page
  1. System Design
  2. Governance

WG: Reliability Scoring Method

Goal: To ensure that the scoring method fairly represents the reliability of the research scored by it.

Critical success factors:

  • A simple, understandable score (e.g. from 1-100).

  • A clear explanation of how the score was arrived at using the CAWE

Necessary conditions and actions:

  • A process to update the score of papers/research when bounties are claimed

  • A process to update the score of papers/research when a paper has been updated to address identified weaknesses

  • The score is relatively stable over time and across disciplines

  • An automated process to update the score of previously scored papers when the CAWE or score methodology is updated

  • Permit additional scoring approaches allowing for more nuanced evaluations

PreviousWorking Group (WG): CRWENextWG: AI Agents

Last updated 10 months ago