#280 2013 · Center for Open Science / Cortex journal (Chris Chambers) · Academic publishing / research integrity
A journal stopped judging papers by their results and started approving them before anyone knew what the results would be
the problem
Evaluators judging work by its outcome quietly rewarded people for engineering outcomes, not for doing the work well
background
Academic journals traditionally decide whether to publish a study after seeing its results, rewarding novel, statistically significant findings and routinely rejecting rigorous studies that found nothing notable. This created a quiet but powerful incentive for researchers to engage in p-hacking (running many analyses until one crosses a significance threshold), selectively reporting only favorable comparisons, or even reframing an exploratory finding as though it had been the original hypothesis all along (HARKing) — because a null result, however carefully obtained, was far less publishable than a flashy one.
Everyone in the field knew this incentive existed and that it was distorting the published scientific record, but the standard fixes — replication requirements, statistical training, ethics guidelines — all operated after the fact, on research that had already been shaped by the pressure to produce a publishable result before a single data point was collected.
what everyone would do
The standard fixes for research integrity problems were all after the fact — replication requirements, statistical training, ethics guidelines, more rigorous peer review of the finished paper — every one of them operating on research that had already been shaped, before a single data point was collected, by researchers' knowledge that the eventual publication decision would hinge on how exciting the result turned out to be.
what they saw
Chambers saw that the corrupting incentive wasn't happening at the review stage at all, it was happening upstream, the moment a researcher designed a study already knowing the outcome would determine publishability. The fix had to move the actual acceptance decision to before the outcome existed, evaluating only the question and the method, which removes the incentive to p-hack, selectively report, or retrofit a hypothesis to match whatever result came out.
the move
In 2013, neuroscientist Chris Chambers introduced Registered Reports at the journal Cortex: a two-stage review format where researchers submit their research question, hypotheses and full methodology for peer review before collecting any data, and the journal grants 'in-principle acceptance' based purely on the importance of the question and the rigor of the design. Stage two, after the study runs, checks only that the pre-registered protocol was actually followed — the paper cannot be rejected for producing a null or unexciting result.
why it works
Submitting the research question, hypotheses and full methodology for peer review before collecting any data, and granting in-principle acceptance based purely on the importance of the question and rigor of the design, locks in the publication decision before the data exists — so there's no longer any incentive to run many analyses until one clears significance, selectively report favorable comparisons, or retrofit an exploratory finding as though it were the original hypothesis, since none of those tactics can change a decision already made. Because stage-two review checks only that the pre-registered protocol was actually followed, a null or unexciting result cannot be rejected for being unexciting, removing the entire selection pressure that made outcome-manipulation rational in the first place. The resulting published record reflects what research actually finds rather than what researchers were incentivized to make it look like they found, which is exactly what the stark gap between 96% significant results in conventional publishing and 44% in Registered Reports demonstrates — most of that gap is a selection artifact, not a difference in what was actually discovered.
the payoff
More than 300 journals have since adopted the Registered Reports format, and the mechanism visibly changed what gets published: roughly 96% of conventionally published psychology papers report a statistically significant result, compared to only about 44% of Registered Reports' pre-registered hypotheses — evidence that the traditional format wasn't just capturing more true positives, it was filtering out a large share of the negative results actually being found.
where it breaks
The mechanism only works for research where the full methodology can genuinely be specified and locked in advance — exploratory, hypothesis-generating research that doesn't yet have a clear question or method to pre-register can't be evaluated this way, and forcing all research into a pre-registered format would eliminate a legitimate mode of discovery. It also depends on reviewers and editors genuinely willing to grant acceptance based on design quality alone, resisting the pull to informally weigh how interesting the eventual finding might be, which requires real institutional discipline to sustain rather than smuggling outcome-based judgment back in at stage two. And it requires researchers willing to accept the loss of flexibility that comes with locking in an approach before starting — a field or individual researchers who prefer keeping analytical flexibility to adapt to what the data show would resist adopting the format even where it's available.
what came after
Registered Reports are now widely cited in the open-science and replication-crisis literature as the structural fix that direct appeals to researcher integrity couldn't achieve, and major journals including Nature expanded their own Registered Reports programs in the years since, treating pre-commitment to method over outcome as a durable publishing standard rather than a niche experiment.
references
- [1]Registered reportWikipedia, 2026en.wikipedia.org
- [2]Registered ReportsCenter for Open Science, 2025cos.io
- [3]Registered reports: an early example and analysisPeerJ (via PubMed Central), 2019pmc.ncbi.nlm.nih.gov