The Reproducibility Crisis Isn't New, But This Way of Fixing It Is
The Reproducibility Crisis Isn't New, But This Way of Fixing It Is
My recent talk with Dr. Jelani Clarke illuminates exactly how this new way of doing science can address the reproducibility crisis, among other academic issues.
Most researchers are familiar with the reproducibility crisis, or the issues regarding inadequate replication of scientific studies. The issue has received more widespread recognition in recent years, encouraging the formation of various meta-research groups and organisations like ReproducibiliTea or Reproducibility for Everyone (R4E). However, most discussions around this issue still don’t include one potentially relevant solution, decentralised science (DeSci).
This worldwide movement aims to transform how scientific research gets done, but many researchers remain largely unaware of it. As an example, I recently attended a colloquium talk at my local university on the replication and credibility crisis in academia. Despite the room being filled with scientists eager to tackle the personal and institutional challenges involved, no one in the room had ever heard of DeSci.
In this edition, I’ll dive into the factors contributing the reproducibility crisis, and what’s being done to address it, as well as how DeSci may contribute to the solution. I’ll pull on the insider perspective from DeSci World lead, Dr. Jelani Clarke.
“We have a replication crisis. 80% of publications can’t be reproduced, even from the original lab that did it.” - Dr. Jelani Clarke, DeSci World
Reproducibility
The issue with reproducibility in scientific research is so prevalent, it has its own name: the replication crisis. Not only is most research not reproduced — meaning that scientists aren’t incentivised to replicate previous findings — but that the vast majority is not reproducible at all. For example, somewhere between 75%-90% of preclinical research is irreproducible. Even more surprising is that this holds true even for the original researchers who conducted the experiment; the original authors often can’t reproduce the same results either.
Why is this an issue? Reproducibility is an integral part of the scientific method. Without independent groups confirming the same results, their reliability and accuracy simply can’t be relied on.
The reproducibility crisis is nothing new, though, and many researchers have already taken matters into their own hands. That’s why the grassroots journal club, ReproducibiliTea, is taking hold in universities across the world. Organisations focused on meta-research also try to address the issue, like the Meta-Research Innovation Center at Stanford or the Association for Interdisciplinary Meta-research and Open Science (AIMOS).
What’s DeSci?
DeSci stands for Decentralised Science, a worldwide movement to change the way scientific research gets done by using blockchain technology. The name itself may be familiar if you’ve already heard of DeFi (decentralised finance) or DeSoc (decentralised society). These concepts rely on the secure and publicly-accessible blockchain (a decentralised ledger) to solve existing issues in finance, society, and now, science.
Check out this previous issue of The Scoop to learn more about what DeSci is.
Fixing reproducibility
Reproducibility isn’t a stand-alone issue. It’s tightly connected to the entire research process. That includes how we incentivise certain research projects over others, how funding is acquired and so on. We’ll cover the three key factors when it comes to remedying the situation and their potential solutions. These include:
Incentivising research that focuses on reproducibility
Funding replication research
Improved data management
Want to learn how to tackle reproducibility better? Join the upcoming AIMOS webinar for dozens of tips from the meta-research community.
Issue 1: Incentives to replicate research
The current system for rewarding scientific contributions doesn’t reward replication. The most obvious symptom of this is the fact that replication studies are rarely cited as well as original works. With the importance of citations in evaluating a researcher’s output, it’s more advantageous for a researcher’s career to continuously pursue new, novel research.
Existing solutions here often have a grassroots aspect to them, such as the variety of institutions and communities taking responsibility for reproducibility. One example is the Institute for Replication’s “Games”, workshops across the world where researchers gather to collaborate on replicating papers. Like most of the other existing initiatives in this area, they rely on voluntary and cultural changes to better incentivise replication.
DeSci approaches attempt to address the underlying incentive issue. One example is ResearchHub, a token-based incentive mechanism that rewards users who share, curate and discuss research. The platform is driven by ResearchCoin, which gives users more influence, like contributing to decisions regarding research directions. Instead of journals, they rely on hubs, which aim to streamline the dissemination of research and knowledge. Additionally, these same principles are also used to encourage peer-review and other kinds of collaboration across the platform.
Issue 2: Funding
The current funding system just isn’t set up to make replication studies appealing for scientists. Cutting edge, novel research is often prioritised, making it hard to get grant funding for replication studies. The DeSci solution here is to incentivise replication studies using blockchain funding mechanisms. This idea applies to more than just replication though, and is the same concept that drives DeSci’s broader mission to fund underfunded science in general.
Decentralised autonomous organisations (DAOs) act as funding agents in the DeSci system. VitaDAO is one of the best-known examples today. With a focus on longevity research, they’ve funded $4 million in over a dozen research projects. One of their projects, the first non-surgical medical approach to periodontitis, is funded as an IP-NFT. By providing a secure record of IP and R&D rights for a project, IP-NFTs ensure researchers receive funds.
Getting funding also means effectively connecting researchers and funders. That’s the objective of centralised online marketplaces like Molecule. Funding through these platforms may be faster than the process of applying for grants. They may also enable additional flexibility in funding traditionally underfunded subjects (say, rare diseases). Other organisations try to do the same beyond the biomedical field, like ScienceFund.
However, the DeSci funding mechanisms may also come with their own potential disadvantages. With reliance on crowdfunding from the general public, it’s unclear how less widely popular topics (non-health) may generate enough interest. Case in point: the vast majority of DAOs are in the biomedical domain. How will DAOs get funding when the research is more theoretical, or without any mass appeal? Not to mention, as with any new technology, it can be difficult to distinguish the genuine players. Take the NFT-based Planck project, for example, which promised to spend 90% of its $24,000 proceeds on a replication study in 2021, but hasn’t released any updates since.
Issue 3: Data management
Access to data and software is a critical first-step in replicating research. Thankfully, this has become more prevalent since the establishment of FAIR data principles and Open Data strategy that work with Open Access publications. For example, the Open Science Foundation’s (OSF) database hosts preprints and data. One potential limitation is that these are published before peer-review. So, researchers may want to wait for official publication before diving into reproducibility work.
Other organisations address this issue from within the existing publishing paradigm, like Code Ocean. This online platforms allows researchers post their projects in capsules. It allows authors to work with reviewers and publishers, so they can easily run the programs of journal submissions. It aims to be “first and foremost, a platform for computational reproducibility”. However, a potential drawback of the platform is resource limitations. Although relatively generous (10 compute hours and 200GB per month), these drop down to 5 minutes and 1GB for non-academics.
The DeSci projects in this area are generally still too new to evaluate, yet. Open repositories like the OpSci Commons exist, but feature limited data thus far. Other groups, like SCINET, are designing decentralised research ecosystems where researchers can manage the entire research lifecycle within the platform. However, SCINET is still in its early stages as well.
The key distinguishing factor in these approaches is the reliance not on traditional cloud storage, which makes communities vulnerable to rising storage costs, but instead on a decentralised network. For example, OpSci relies on FileCoin. Its decentralised nature makes the data more reliably archived and accessed compared to traditional storage options. Interestingly, the venture also requires that data passes various automated validations to ensure it meets reproducibility standards.
Unlike traditional approaches, DeSci solutions tend to address the reproducibility crisis from multiple angles at once, rather than splitting and solving these issues separately. However, much is still theoretical and not yet tested in the wider scientific community.
The success of some DeSci organisations so far indicates a promising future. By bringing DeSci to the forefront of discussions on academic issues like publishing, funding, and reproducibility, we can explore a whole new approach for solving these long-standing issues in research.
Thank you to Dr. Jelani Clarke from DeSci World for his contributions to this article. We are always seeking experts to interview for The Scoop. If you’re interested, please let us know.
Resources
AIMOS: Tip talk, August 23, 2023 Online Event on reproducibility tips
Capabilities of SCINET in DeSci, Integrating Scientific Research with Blockchain, Aug 2023
Decentralized Science: The New Opportunity for Research Data Management, Aug 2023
The replication crisis has led to positive structural, procedural, and community changes, July 2023
User incentives for blockchain-based data sharing platforms, April 2022
Why Most Published Research Findings Are False, Ioannidis 2015
Framework for Open and Reproducible Research Training (FORRT)
Reproducibility for Everyone (R4E)
The Institute for Replication (I4R)
Association for Interdisciplinary Meta-research and Open Science (AIMOS)