How PaperScore Works

PaperScore accepts articles from all disciplines, particularly from interdisciplinary and multidisciplinary studies. It does not categorize articles, but rather tags them with keywords.

Instead of an editorial board, PaperScore uses a collective intelligence algorithm that is implemented in a computer code and controls the peer-review process with minimal human intervention.

After the peer-review process, the authors see the reviewers’ comments and median scores, and then can decide whether to publish their manuscript or not. If published, it will be indexed in several databases such as Scopus, PubMed and Google Scholar, and will be accessible by all universities and academic institutions around the world. It can also become an NFT (ERC 1155) on the Ethereum blockchain.

Whether the authors decide to have their manuscript published or not, they can receive valuable feedback and comments from the experts who reviewed it. This helps them to improve their manuscript and submit it to other journals if they want to.

Go to Guidelines page

Entrenched Incumbents

The current research publication industry is dominated by a three-sided network effect between authors, institutions, and journals. Institutions and authors want to publish in well-known journals, while journals want to publish works of well-known authors (and institutions) who have published in well-known journals. This can lead to self-fulfilling prophecies. PaperScore aims to break this outdated ecosystem by completely decentralizing the peer-review process.

Moreover, the current ecosystem gives editors the authority to select reviewers or reject a submission without any peer-review. They also make the final decision on each submission. But like all people, editors are also susceptible to biases. As a result, journals are subject to a number of editorial biases. Such biases are reinforced if the editors are always succeeded by those who have published in the journal before.

Publication Bias occurs when studies that find no significant effects are not published, even though the results were obtained rigorously. Discipline Bias occurs when multidisciplinary and interdisciplinary studies are rejected for not fitting in the particular study areas of the major journals. PaperScore provides a home for such homeless studies.

PaperScore Review ProcessReview Process

Once an author submits a manuscript to PaperScore, the backend code randomly selects up to five potential reviewers based on the manuscript’s keywords and citations. Each potential reviewer receives a message and decides whether or not to evaluate the manuscript. Reviewers also have the option to refer manuscripts to other potential reviewers they deem to be experts in the field. The next reviewer will also have the same option.Once an author submits a manuscript to PaperScore, the backend code randomly selects up to five potential reviewers based on the manuscript’s keywords and citations. Each potential reviewer receives a message and decides whether or not to evaluate the manuscript. Reviewers also have the option to refer manuscripts to other potential reviewers they deem to be experts in the field. The next reviewer will also have the same option. Based on the small-world phenomenon, each chain will get to a suitable reviewer in fewer than six referrals on average. Most likely, it will be fewer than three referrals because of the purposeful randomization that incorporates relevant information such as keywords.

Based on the small-world phenomenon, each chain will get to a suitable reviewer in fewer than six referrals on average. Most likely, it will be fewer then three referrals because of the purposeful randomization that incorporates relevant information such as keywords.Currently, the PaperScore review process is double-blind and is controlled by the scaffolder (the Editor In Chief); the scaffolder will try to find the best reviewers for each manuscript and calibrates the rating scores to improve their accuracy and reliability. In the future, however, the scaffolder will be out of the loop and the process will be fully automated and triple-blind, moving towards a neutral meritocracy.

Currently, the PaperScore review process is double-blind and is controlled by the scaffolder (the Editor In Chief); the scaffolder will try to find the best reviewers for each manuscript and calibrates the rating scores to improve their accuracy and reliability. In the future, however, the scaffolder will be out of the loop and the process will be fully automated and triple-blind, moving towards a neutral meritocracy.

Rating Scores

Each reviewer can anonymously rate a manuscript on four dimensions:

Writing & Composition
Rigor & Validity
Novelty & Contribution
Importance & Value

These dimensions are defined in theScore Guidelines. Reviewers can also leave anonymous comments to support their rating scores. Once a sufficient number of reviewers submit their rating scores, the median score for each dimension is calculated and reported to the author(s). Author(s) can only see the anonymous reviewer comments and the aggregated median scores, but not the individual scores, or the names of the reviewers.

Instead of accepting or rejecting, PaperScore amends each manuscript with comments and scores. This provides valuable information for authors and potential readers.

Publication Decision

Upon seeing the anonymous comments and median scores, the author(s) can decide to release, revise, or withdraw the manuscript. If they decide to release the manuscript, it will be published with its anonymous comments and median scores. However, if they are not happy with the review results, they can choose to revise and resubmit the manuscript; the manuscript will then undergo the same review process obtaining a new set of rating scores and comments.

Therefore, every submitted manuscript that follows the essential formatting requirements, such as anonymity, will be reviewed and then published if, and only if, the corresponding author decides to release it. Since PaperScore is fully electronic, it is not limited to the traditional space limitations of the paper-based journals and can include thousands of papers in its database. To find the most relevant and high-quality studies, the readers can search, filter, and sort papers based on keywords, authors, dates, and, most importantly, median scores.

Reviewer Points

Most of the traditional journals rely on anonymous reviewers whose performance is only visible to the editors of that journal. Therefore, the reviewers spend time evaluating the manuscripts based on the hope that the editor remembers their contributions. However, this provides little incentive for meticulous and thorough reviews in time. PaperScore solves this problem by defining reviewer points.

The PaperScore code automatically accumulates and calculates reviewer points for reviewers and referrers. For each submission, the median reviewer(s) for each dimension will earn reviewer points. The median reviewers are those whose rating scores are closest to the median score for each dimension. Also, the referrers who referred the manuscript to those median reviewers receive partial points for their accurate referrals. The relative amount of reviewer points for accurate rating and accurate referral can be fine-tuned through an RSM algorithm.

While the reviewers and their referrers are kept anonymous for every manuscript, the aggregated reviewer points for reviewers are publicly visible on their profiles. Also, the top reviewer points will be listed on the PaperScore homepage to publicize the best reviewers. The reviewer points will give the reviewers credit for their contributions and incentivize for better reviews and referrals.

The Vision

In the future, PaperScore will utilize Artificial Intelligence to assess if a submitted manuscript is anonymous and follows the required formatting and style standards. Also, AI can perform part of the evaluation and provide a score for Writing & Composition. Furthermore, a deep learning algorithm will be applied to continuously improve the review process based on the quality of the reviews, feedback, and forward citations.

Eventually, PaperScore will be implemented as a DApp on a generalized Blockchain like Ethereum or Solana, whereby a smart contract executes the review process and accumulates the rating scores and reviewer points. In time, PaperScore will become an independent self-governed platform to evaluate and publish research findings, and to establish reputation and credibility for scholars and researchers around the world.

* This is an attempt to explain the general purpose of PaperScore and is subject to change. Parts of the above mentioned processes and the methods used in this software are included in the US patent application 63151798 (pending patent): "Method and Apparatus for Decentralized Evaluation of Papers, Articles, Artifacts, and Designs" (Inventor: Hamed Khaledi)