Everyday Scientist

The Case for Preregistration in Cell Biology: A Practical Guide

The basics of the scientific method—hypothesize, test, revise—rely on prediction. By forming a hypothesis only after seeing the data, we fail to severely test our ideas. I encourage cell biologists to preregister their study designs and statistical analyses. I also provide practical tutorials for making this a reality.

Preregistration will Bolster Cell Biology

Preregistration has helped add rigor to various fields, including psychology and clinical trials (Nosek et al., 2018). But in cell biology, molecular biology, and other basic research, preregistration has not become popular, at least partially due to a general belief that it would be overly burdensome and impractical in exploratory experiments. Here, I attempt to dispel those myths and provide practical steps for cell biologists to preregister their experiments and data analysis. Instead of hindering research activity, preregistration can aid experimentalists in designing better experiments, reducing waste, and adding rigor to their science.

While preregistration may sound daunting, it need not be. Most of us already write down a basic experimental plan in our lab notebook before we step into the lab, so uploading that to a repository (public or private) is a small effort. But that simple gesture could encourage researchers to flesh out their plans and more carefully consider potential pitfalls, like not having a sufficient sample size. There is nothing more annoying than completing weeks in the lab only to realize that you forgot an essential control or really needed an additional replicate to get a statistically significant result. Better planning can help avoid wasted effort, and preregistration helps. Getting feedback on your plan with labmates and colleagues is easier with a formal plan, and preregistration ensures that the PI and all the researchers on the project are on the same page.

We all know that arranging the data analysis procedure before seeing the results is good scientific hygiene. Preregistration forces us to make it a habit to plan the analysis, thereby reducing the temptation to p- and N-hack our way to “significance.” Furthermore, listing a priori plans and any deviations in the final paper adds transparency and trust to scientific publishing. Finally, committing to disclose results before the outcome is known helps alleviate the problem of selective reporting in scientific publishing.

Ancillary Benefits

Taking the time to plan an adequate study design before stepping into the lab is of course beneficial in itself. But preregistering an analysis plan could actually increase statistical power and even reduce the number of required samples. For instance, if the hypothesis predicts a directional effect of a treatment, researchers can plan for a one-sided t-test, which requires fewer samples to achieve the same power (Lakens, 2022). While it wouldn’t be fair to opt for a one-sided test after collecting the data and learning the direction of the effect, documenting the bold prediction of a directional effect beforehand is in some ways superior to the standard two-sided test. (Convincing reviewers of this may be a challenge, but they should look favorably on the transparency and rigor of preregistration.)

Preregistration can further reduce the burden on cell biologists by allowing them to “peek” at their data partway through completion of an experiment, providing the opportunity to stop data collection with fewer replicates if the effect is sufficiently large. “N-hacking” is when researchers arbitrarily add samples until they reach a “significant” result, but with proper planning using sequential analysis—correcting the α level for multiple peeks—one can possibly stop early while maintaining the desired false positive rate (Lakens, 2022; Schott et al., 2019). A major benefit of preregistering a sequential analysis is that the expected number of biological replicates is actually smaller than the maximum sample size you would plan, because peeking allows the possibility of stopping the experiment early with an already significant p-value (Lakens, 2022). This is a common approach in clinical trials, and cell biologists could benefit from borrowing this statistically rigorous way to make our lives easier. (Note that at very small sample sizes, like n=3 per stage per condition, corrections like Pocock have inflated false-positive rates. Rom-McTague (Rom and McTague, 2020) is a modified Pocock correction for small n.)

Finally, preregistering a plan can help make null results more credible to editors and reviewers, making it easier to publish all your results. Communicating the full picture, even the experiments that didn’t match your expectations, strengthens the integrity of your science by allowing you to tell the full story. Unfortunately, journals often make it (unofficially) difficult to publish results that do not have statistical significance or are otherwise ambiguous. Preregistration gives authors ammunition to justify including null results in their published findings.

Encouraging Preregistration

Doing robust science is a reward in itself, and the vast majority of researchers already strive to do science right. If preregistration became a norm—just like displaying scale bars on microscopy images, providing raw data, uploading structural data to PDB, sharing code on GitHub, reporting sample sizes, including loading controls for Westerns, and showing uncropped gels in the supporting information—then scientists would eventually preregister as a habit.  

But some external encouragement couldn’t hurt. Journals and reviewers could ask for preregistration documentation, and reward authors who can produce such plans. For example, when reviewer 2 asks for additional experiments or more replicates, editors should be emboldened by a preregistration to politely disregard such demands (when they are indeed unreasonable). Furthermore, editors should feel more willing to accept negative or ambiguous results when the experiment was preregistered. And journals should prominently display a link or badge on papers that provide preregistration documentation.

A Practical Guide to Preregistering

The OSF repository provides resources for recording preregistration reports (Open Science Framework, 2023), including what to do when plans change, and others have expanded on the benefits and practical application of preregistration in small studies (Johnson and Cook, 2019). Here, we provide a crude template (Box 1) for a simple preregistration. See Box 2 for example paragraphs describing the preregistration in the final manuscript. As your plans develop while you traverse a project, you can augment the preregistered plan with additional experiments. Because most cell biology papers include several smaller experiments, the preregistered plan can include descriptions of multiple experiments, or you may opt to submit a separate short plan for each experiment.

Common Objections

I don’t want my competitors knowing my experiments before I even start.

Preregistrations need not be public to be effective. Depositing a preregistration privately (or even simply writing it down, timestamping it, and storing it on a local computer) still keeps researchers accountable and the research design more robust against post-hoc statistical exploration. The plan can eventually be made public to peer reviewers and readers of the ultimate publication. That said, making a plan public—or at least sending it to some trusted colleagues—offers an opportunity to get helpful feedback on an experiment before you even enter the lab.

My experiments are too exploratory to plan ahead.

It probably doesn’t make much sense to preregister pilot studies and research that is still in the beginning, exploratory stages. So maybe 90% of the work you do in the lab—culturing and observing cells, developing assays, purifying proteins, making and characterizing reagents, performing chemical analyses, and running pilot experiments—doesn’t need a preregistration. But before you sit down to start an actual experiment, it is worth planning: write down, discuss, and preregister how you will tackle the project. It is challenging to recognize precisely when research shifts from exploratory to a confirmatory experiment, and I don’t have a perfect prescription for identifying which work deserves preregistration, but most scientists know in their gut when their efforts are destined for a figure in a paper. When in doubt, just fill out the preregistration template (Box 1); you can always discard it if it turns out it wasn’t needed or update it as necessary as the project progresses. That shouldn’t take too much time, and it might help focus your attention on deficits in your plan early on.

This is too much work.

Preregistering adds little administrative burden, since the hypothesis, research plan, and often even the statistical analyses have already been laid out in research grant applications, lab notebooks, or thesis committee presentations. But taking the simple steps of formally organizing and uploading these plans to a repository ensures you have a clear strategy and helps avoid common pitfalls of poor experimental planning. Each experiment can be preregistered on its own, so there’s no need to decide what the ultimate paper will be during the early planning stages.

This will lock me into a plan that might send me down the wrong path.

Preregistration is a plan, not a prison (Open Science Framework, 2023). If plans change before collecting the data, it’s as simple as updating the preregistered documents. If you perform unplanned tests or data analysis during or after the experiment, you can transparently report deviations from the plan and justifications in the final paper (Lakens, 2024). If you make an unexpected discovery or do additional experiments not listed in your original preregistration, you can either update the preregistration or simply note in the final manuscript that that particular experiment had not been listed in the original preregistered plan. This adds transparency and empowers the reader with useful information. “Having  preregistered  your  study  does  not  prevent  you from  publishing  any  analysis  you  believe  is  interesting  or informative. All that preregistration does is clarify what tests were formulated a priori and what tests were not” (Forstmeier et al., 2017).

What if the experiment doesn’t give the result I expected? Am I committed to trying to publish negative results?

Preregistration doesn’t obligate you to publish every result: you’re still free to tell the story the way you wish. The difference is that, when you document your full process, the readers will know that the findings are not cherry-picked, adding credibility and transparency to whatever you ultimately publish.

My Personal Experience with Preregistration

This is where I am seeking input from anyone who has ever preregistered their experiments before. For my next experiment, I will preregister the plan and the analysis approach and include that when we submit to a journal. But that will be months away. Has anyone already done this and want to expound upon the process and the outcome?

Conclusion

The value of preregistration has been demonstrated for years in clinical research and other fields, and it is time to apply this proven technique to cell biology. Of course, preregistration is not magic: it will not singlehandedly solve the replication crisis, eliminate fraud, or make all research findings true. Improving science will take a concerted effort by journals, authors, and funders to better incentivize transparent protocols and more rigorous testing of findings (Forstmeier et al., 2017; Kohrs et al., 2023; Lord et al., 2026). But this small yet powerful step would increase rigor in scientific research and help strengthen the literature. Ultimately, if you completely ditch your preregistered plan and don’t even mention it in your paper, while this is dishonest, it’s no worse off than the status quo. So there’s really no harm in experimenting with preregistration.

References

• Forstmeier, W., E.-J. Wagenmakers, and T.H. Parker. 2017. Detecting and avoiding likely false-positive findings – a practical guide. Biol. Rev. Camb. Philos. Soc. 92:1941–1968. doi:10.1111/brv.12315.
• Johnson, A.H., and B.G. Cook. 2019. Preregistration in Single-Case Design Research. Except. Child. 86:95–112. doi:10.1177/0014402919868529.
• Kohrs, F.E., S. Auer, A. Bannach-Brown, S. Fiedler, T.L. Haven, V. Heise, C. Holman, F. Azevedo, R. Bernard, A. Bleier, N. Bössel, B.P. Cahill, L.J. Castro, A. Ehrenhofer, K. Eichel, M. Frank, C. Frick, M. Friese, A. Gärtner, K. Gierend, and T.L. Weissgerber. 2023. Eleven strategies for making reproducible research and open science training the norm at research institutions. eLife. 12. doi:10.7554/eLife.89736.
• Lakens, D. 2022. Improving Your Statistical Inferences.
• Lakens, D. 2024. When and how to deviate from a preregistration. Collabra: Psychology. 10. doi:10.1525/collabra.117094.
• Lord, S.J., A. Charles-Orszag, K. Skruber, R.D. Mullins, and A. Rehfeld. 2026. Peer replication : A new tier of science built on reproducibility. EMBO Rep. doi:10.1038/s44319-026-00705-8.
• Nosek, B.A., C.R. Ebersole, A.C. DeHaven, and D.T. Mellor. 2018. The preregistration revolution. Proc Natl Acad Sci USA. 115:2600–2606. doi:10.1073/pnas.1708274114.
• Open Science Framework. 2023. Preregistration – OSF Support.
• Rom, D.M., and J.A. McTague. 2020. Exact critical values for group sequential designs with small sample sizes. J. Biopharm. Stat. 30:752–764. doi:10.1080/10543406.2020.1730878.
• Schott, E., M. Rhemtulla, and K. Byers-Heinlein. 2019. Should I test more babies? Solutions for transparent data peeking. Infant Behav. Dev. 54:166–176. doi:10.1016/j.infbeh.2018.09.010.

____________________________________________________

____________________________________________________

Box 2: Example paragraphs for eventual manuscript

We preregistered our research plans and statistical analyses (link to plans).

Based on a small pilot experiment, we predicted that abc drug treatment would reduce cell speeds relative to control, so we proposed a one-tailed t-test with α=0.05. Based on the Cohen’s effect size of d=2 from the pilot experiment, we proposed n=8 biological replicates (4 control and 4 treated), which should give a power of around 80%. After the experiment, the p-value was 0.043. However, we noticed after the data was collected that there was substantial variation in the speed of the control cells on different days. Because control and treated samples were split from the same flask of cells for each day’s experiment, we subsequently concluded that a paired t-test was the more appropriate approach and therefore departed from our preregistered analysis plan. The one-sided paired t-test resulted in p=0.011. Results are shown in Fig 2.

Also, after observing the full results, we noted that the nuclei seemed smaller in the drug-treated samples. Although not in our original preregistered plan, we decided to also compare the size of the nuclei in control vs. treated. We blinded the filenames and measured the area of the nuclei of all the cells in the dataset. The two-tailed paired t-test resulted in a p=0.034. Results are shown in Fig 3.

For the internalization experiment, we predicted that knockdown of gene xyz would abrogate endocytosis. We proposed a one-sided paired t-test with α=0.05 and planned n=6 biological replicates (3 control paired with 3 treated). However, our results showed no obvious trend in the internalization rate and the p-value was nonsignificant at p=0.31. Although the preregistered plan calculated that the experiment was adequately powered to detect a large biologically meaningful effect, the sample-to-sample variance was larger than we had anticipated, so we cannot conclude definitively that knockdown of gene xyz does not alter endocytosis rates, but it does set a floor for the effect size. Future tests of this hypothesis will need to either reduce the experimental noise or increase the number of replicates. This null result is reported in Fig S1.

For the neuron experiment, we proposed using an unpaired design, and we planned to use male mice from the same litter. We proposed to sacrifice control and knockout mice, fix and stain brain tissue, and image the average intensity of xyz marker in 100 neurons from each mouse. Sample collection was performed blinded with respect to the identity of the sample. We proposed a two-tailed t-test to calculate the p-value, because we did not know whether the marker would increase or decrease with gene knockout. To potentially reduce the number of mice needed, and based on an anticipated effect size of approximately 1.5, we planned a two-stage sequential analysis with n=5 mice per stage per condition; to account for multiple assessments of the data, we proposed the Rom-McTague α of 0.0285 for each stage, for a cumulative α of 0.05. After n=5 knockdown and 5 control mice, we calculated p=0.024, so we halted the experiment according to our plan. Results are shown in Fig 5.

end grant writing; award prizes for papers instead

My proposal for funding academic research science: instead of funding grants, just award financial prizes for good papers.

Implementation

Award half the prize upon publication (or preprint); award the second half upon an independent replication of the results, 25% to the original authors and 25% to the replicators. (If a good-faith replication attempt fails, the replicators would still receive their portion, but the original authors would not receive their replication bonus. The measure for the quality of the replication would be whether the work could be published in a reputable journal.)

Prize winning papers would be chosen by rotating committees of professors following a rubric announced prior to the prize committee meeting (replacing grant study sections). Different committees could be assembled for different research topics, with appropriately tailored rubrics. Any paper published in the prior 12 months would be eligible during each prize committee meeting (there could be multiple committees convened each year, giving any given paper multiple chances at winning). Some committees would be assembled to identify “sleeper” papers: works that are up to 5 years old that had been overlooked by previous committees but have proven to be especially valuable.

The prize amount could vary by field or even by paper. A typical R01 grant is $150k+ annually for 5 years. So prizes could vary from $200k up to $1M+ for a major tour-de-force paper with substantial impact. The goal would be to distribute the same pot of money to approximately the same number of labs in roughly the same proportion, but with a much smaller bureaucratic burden and with better incentives for reproducibility.

There would probably need to be some cap on the total prize money an individual PI could receive at any one time. If they are over the cap, then either their papers would simply not be eligible for a prize that round or their prize money would flow to the collaborators on that paper or department.

Prize money would still go to the PI and lab via the university, not as a blank check to an individual professor. Indirects/overhead would still be applied. The goal would be to have approximately the same amount of money flowing to the university and the labs, but with a less onerous process.

Advantages

The prize approach accomplishes the following:

• Rewards actual outputs rather than promises.
• Creates strong incentives for reproducible work.
• Incentivizes impactful, innovative results that are convincing and accepted by the scientific community.
• Frees up professors’ time to focus on paper writing and mentoring instead of grant writing.
• Reduces the workload for university grant offices, which currently spend considerable time reviewing and approving proposals before submission, thereby reducing costs and freeing administrative staff at universities and funding agencies to focus on other support functions.
• Removes back-and-forth about budgets, justifications, biosketches, data-sharing policies, etc.
• Does away with byzantine formatting requirements.
• Obviates¹ the need for progress reports and renewals throughout the grant period.

Professors spend a huge amount of time writing grants, and university grant offices dedicate substantial resources helping researchers navigate complex formatting and compliance requirements, including page lengths, fonts, hyperlinks, sections, etc. Academic researchers currently focus on publishing their work, so grants should reinforce that drive, not distract PIs with hundreds of hours of grant writing. Prize committees would replace grant review panels (roughly equivalent work), while reducing the administrative burden on both university grant offices and granting agencies.

The current grant funding structure purports to be forward-looking, providing funding for good ideas to be implemented. But actual grants require not only a good track record for the PI, but also a large amount of preliminary data. So, in effect, grants fund a lot of work that’s already been completed, and the awarded money usually gets spent on lab work outside the scope of the original proposal. So, while appearing to be prospective, grants are often retrospective in practice. Granting prizes for good papers simply makes official how funding already flows in reality.

How to fund research before winning prizes

The central challenge would be providing runway funding, especially for early-career researchers who have no eligible papers yet. With fewer resources needed for proposal preparation support, some overhead funds could be redirected to departmental slush funds. Departments could then provide startup packages, bridge funding between prizes, or support for labs that demonstrate progress toward publishable work.

This would shift evaluation of proposed research from granting agencies to local departments, where a lab’s potential is more knowable. However, departments are often rife with unhealthy politics and lack sufficient guardrails against favoritism and bias. Whether departmental evaluation would be less burdensome than current grants – and whether freed-up overhead would provide sufficient funding – remains an open question.

Alternatively, granting agencies could dedicate some funds for prizes awarded specifically to a professor’s first paper, or maintain a smaller pot of traditional proposal-style funding for early-career researchers. But as it stands, professors who don’t publish within their first few years are unlikely to receive traditional grants or be awarded tenure anyway, so the prize approach doesn’t necessarily create more pressure than the current system.

Downsides

Not every good paper would win a prize. Yes, of course there would still be disappointing results: inevitably, some deserving papers would fail to attract money because the subject matter is out of fashion or reviewer bias or a host of other unfair reasons. But with a limited amount of money and an effectively unlimited number of professors, it’s impossible to fund everyone, so someone will always be neglected. Winning a prize would be no more unpredictable than receiving a good score on a grant. My proposal is no less fair than the whims of the current granting process.

Incentivizes short-term work. Scientists would be less motivated to tackle a longer-term project, even if it is promising. The department or other funding agency would need to step in to float the project for years until it comes to fruition. And if whoever funds that float requires proposals to have the same level of detail and complexity as current grant proposals, we’d be back to square one. But proposals for future work need not be as onerous as current NIH regulations, especially if funded by the department or by private organizations like HHMI. Another source of funding for longer-term projects may actually come from paper prizes: professors who have won multiple prizes in the recent past would have ample money to fund collaborations and longer projects.

Disincentivizes high-risk research. Researchers might be wary of taking on high-risk/high-reward projects, because, if the experiments fail, they couldn’t expect a prize. That said, if such a project succeeds, then the prize money would likely be larger. Such is the nature of high-risk/high-reward. Therefore, the prize money for impactful work would need to be sufficiently large to encourage at least a portion of researchers to take on projects that are higher risk. Furthermore, if researchers are able to publish details of a failed “moonshot” that advanced the field despite not succeeding, such a paper would be eligible for a prize. The flexibility of prize funding might actually enable more ambitious research, since successful researchers would have discretion over how to deploy their winnings without the constraints of grant-specified aims.

Disincentivizes collaborations. Collaborators would likely negotiate some cut of a potential prize ahead of time, or there would be a standard formula (e.g. 75% to the last author and the remainder of the prize to the other PI collaborators on the paper). Maybe PIs would be less willing to add collaborators to a project, for fear that they may need to share some of the prize money with other PIs. Maybe there could be a bonus to help support collaborators.

Expensive projects. This would not work for huge projects like clinical trials or massive particle physics consortiums, which would need ample funding before the experiments or construction actually begin. But those could continue to be funded in the current fashion.

What will professors do with their freed up time? Part of the job of a professor is to write grants. If they suddenly don’t need to write grants, will they actually spend more time teaching and mentoring, or will they just work less? Not sure. That’s a valid point.

Moving forward

The next step would be to run a pilot. Initially, 20% of NIH and NSF funding could be converted to this model in select fields that would be amenable, then evaluated after 5 years. At that time, the percentage of funding provided via the prize mechanism might be increased or decreased based on the outcome of the pilot. It’s unlikely that granting agencies would ever completely eliminate the traditional grant proposal, but the health of academic science could be strengthened if even a minority of the funding rewarded impactful, reproducible papers.

___________________

1. Thesaurus, don’t fail me now. ↩︎

eLife’s new publishing policy

Let me preface this post with the admission that I’m likely wrong about my concerns about eLife’s new model. I was actually opposed to preprints when I first heard about them in 2006!

The Journal eLife and Mike Eisen announced it’s new model for publishing papers:

• Authors post preprint.
• Authors submit preprint to eLife.
• eLife editorial board decides whether to reject the manuscript or send out for review.
• After reviews, the paper will be published no matter what the reviews say. The reviews and an eLife Assessment will be published alongside the paper. At this point, the paper has a DOI and is citable.
• Authors then have a choice of whether to revise their manuscript or just publish as-is.
• When the authors decide the paper is finalized, that will become the “Version of Record” and the paper will be indexed on Pubmed.

Very interesting and bold move. The goal is to make eLife and its peer review not a gatekeeper of truth, but instead a system of evaluating and summarizing papers. Mike Eisen hopes that readers will see “eLife” and no longer think “that’s probably good science” and instead think “oh, I should read the reviews to see if that’s a good paper.”

Potential problems

But here are my primary concerns

1. This puts even more power in the hands of the editors to effectively accept/reject papers. And this process is currently opaque, bias-laden, and authors have no recourse when editors make bad decisions.
2. The idea that the eLife label will no longer have prestige is naive. The journal has built a strong reputation as a great alternative to the glam journal (Science, Nature, Cell) and that’s not going away. For example, people brag when their papers are reviewed in F1000, and I think the same will apply to eLife Assessments: readers will automatically assume that a paper published in eLife is high-impact, regardless of what the Assessment says.
3. The value that eLife is adding to the process is diminishing, and the price tag is steep ($2000).
4. The primary problem I have with peer review is that it is simultaneously overly burdensome and not sufficiently rigorous. This model doesn’t substantially reduce the burden on authors to jump through hoops held by the reviewers (or risk a bad eLife Assessment). It also is less rigorous by lowering the bar to “publication.”

Solutions

Concern #1: I think it’s a step in the wrong direction to grant editors even more power. Over the years, editors haven’t exactly proven themselves to be ideal gatekeepers. How can we ensure that the editors will act fairly and don’t get attracted by shiny objects? That said, this policy might actually put more of a spotlight on the desk-rejection step and yield change. eLife could address this concern in various ways:

• The selection process could be a lottery (granted, this isn’t ideal because finding editors and reviewers for a crappy preprint will be hard).
• Editors could be required to apply a checklist or algorithmic selection process.
• The editorial process could be made transparent by publishing the desk rejection/acceptace along with the reasons.

Concern #2 might resolve itself with time. Dunno. Hard to predict how sentiment will change. But I do worry that eLife is trying to change the entire system, while failing to modify any of the perverse incentives that drive the problems in the first place. But maybe it’s better to try something than to do nothing.

Concern #3 is real, but I’m sure that Mike Eisen would love it if a bunch of other journals adopted this model as well and introduced competition. And honestly, collating and publishing all the reviews and writing a summary assessment of the paper is more than what most journals do now.

Journals should be better gatekeepers

But #4 is pretty serious. The peer review process has always had to balance being sufficiently rigorous to avoid publishing junk science with the need to disseminate new information on a reasonable timescale. Now that preprinting is widely accepted and distributing results immediately is super easy, I am less concerned with latter. I believe that the new role of journals should be as more exacting gatekeepers. But it feels like eLife’s policy was crafted exclusively by editors and authors to give themselves more control, reduce the burden for authors, and shirk the responsibility of producing and vetting good science.

There are simply too many low-quality papers. The general public, naive to the vagaries of scientific publishing, often take “peer-reviewed” papers as being true, which is partially why we have a booming supplement industry. Most published research findings are false. Most papers cannot be replicated. Far too many papers rely on pseudoreplication to get low p-values or fail to show multiple biological replicates. And when was the last time you read a paper where the authors blinded their data acquisition or analysis?

For these reasons, I think that the role of a journal in the age of preprints is to better weed out low-quality science. At minimum, editors and peer reviewers should ensure that authors followed the 3Rs (randomize, reduce bias, repeat) before publishing. And there should be a rigorous checklist to ensure that the basics of the scientific process were followed.

Personally, I think the greatest “value-add” that journals could offer would be to arrange a convincing replication of the findings before publishing (peer replication), then just do away with the annoying peer review dog-and-pony show altogether.

Conclusion

We’ll have to wait and see how this new model plays out, and how eLife corrects stumbling blocks along the way. I have hope that, with good editorial team and good practices/rules around the selection process, eLife might be able to pull this off. Not sure if it’s a model that will scale to other, less trustworthy journals.

But just because this isn’t my personal favorite solution to the problem of scientific publishing, that doesn’t mean that eLife’s efforts won’t help make a better world. I changed my mind about the value of preprints, and I’ll be happy to change my mind about eLife’s new publishing model if it turns out to be a net good!

Replace Peer Review with “Peer Replication”

UPDATE: EMBO Reports article here and a write-up in Nature here.

As I’ve posted before and many others have noted, there is a serious problem with lack of adequate replication in many fields of science. The current peer review process is a dreadful combination of being both very fallible and also a huge hurdle to communicating important science.

Instead of waiting for a few experts in the field to read and apply their stamp of approval to a manuscript, the real test of a paper should be the ability to reproduce its findings in the real world. (As Andy York has pointed out, the best test of a new method is not a peer reviewing your paper, but a peer actually using your technique.) But almost no published papers are subsequently replicated by independent labs, because there is very little incentive for anyone to spend time and resources testing an already published finding. That is precisely the opposite of how science should ideally operate.

Let’s Replace Traditional Peer Review with “Peer Replication”

Instead of sending out a manuscript to anonymous referees to read and review, preprints should be sent to other labs to actually replicate the findings. Once the key findings are replicated, the manuscript would be accepted and published.

(Of course, as many of us do with preprints, authors can solicit comments from colleagues and revise a manuscript based on that feedback. The difference is that editors would neither seek such feedback nor require revisions.)

Along with the original data, the results of the attempted replication would be presented, for example as a table that includes which reagents/techniques were identical. The more parameters that are different between the original experiment and the replication, the more robust the ultimate finding if the referees get similar results.

Incentives

What incentive would any professor have to volunteer their time (or their trainees’ time) to try to reproduce someone else’s experiment? Simple: credit. Traditional peer review requires a lot of time and effort to do well, but with zero reward except a warm fuzzy feeling (if that). For papers published after peer replication, the names of researchers who undertook the replication work will be included in the published paper (on a separate line). Unlike peer review, the referees will actually receive compensation for their work in the form of citations and another paper to include on their CV.

Why would authors be willing to have their precious findings put through the wringer of real-world replication? First and foremost, because most scientists value finding truth, and would love to show that their findings hold up even after rigorous testing. Secondly, the process should actually be more rewarding than traditional peer review, which puts a huge burden on the authors to perform additional experiments and defend their work against armchair reviewers. Peer replication turns the process on its head: the referees would do the work of defending the manuscript’s findings.

Feasible Experiments

There are serious impediments to actually reproducing a lot of findings that use seriously advanced scientific techniques or require long times or a lot of resources (e.g. mouse work). It will be the job of editors—in collaboration with the authors and referees—to determine the set of experiments that will be undertaken, balancing rigor and feasibility. Of course, this might leave some of the most complex experiments unreplicated, but then it would be up to the readers to decide for themselves how to judge the paper as a whole.

What if all the experiments in the paper are too complicated to replicate? Then you can submit to JOOT.

Ancillary Benefits

Peer replication transforms the adversarial process of peer review into a cooperation among colleagues to seek the truth. Another set of eyes and brains on an experiment could introduce additional controls or alternative experimental approaches that would bolster the original finding.

This approach also encourages sharing experimental procedures among labs in a manner that can foster future collaborations, inspire novel approaches, and train students and postdocs in a wider range of techniques. Too often, valuable hands-on knowledge is sequestered in individual labs; peer replication would offer an avenue to disseminate those skills.

Peer replication would reduce fraud. Often, the other authors on an ultimately retracted paper only later discover that their coworker fabricated data. It would be nearly impossible for a researcher to pass off fabricated data or manipulated images as real if other researchers actually attempt to reproduce the experimental results. 

Potential Problems

One serious problem with peer replication is the additional time it may take between submission and ultimate publication. On the other hand, it often takes many months to go through the traditional peer review process, and replicating experiments may not actually add any time in many cases. Still this could be mitigated by authors submitting segments of stories as they go. Instead of waiting until the entire manuscript is polished, authors or editors could start arranging replications while the manuscript is still in preparation. Ideally, there would even be a  journal-blind mechanism (like ReviewCommons) to arrange reproducing these piecewise findings.

Another problem is what to do when the replications fail. There would still need to be a judgement call as to whether the failed replication is essential to the manuscript and/or if the attempt at replication was adequately undertaken. Going a second round at attempting a replication may be warranted, but editors would have to be wary of just repeating until something works and then stopping. Pre-registering the replication plan could help with that. Also, including details of the failed replications in the published paper would be a must.

Finally, there would still be the problem of authors “shopping” their manuscript. If the replications fail and the manuscript is rejected, the authors could simply submit to another journal. I think the rejected papers would need to be archived in some fashion to maintain transparency and accountability. This would also allow some mechanism for the peer replicators to get credit for their efforts.

Summary of Roles:

• Editor:
  • Screen submissions and reject manuscripts with obviously flawed science, experiments not worth replicating, essential controls missing, or seriously boring results.
  • Find appropriate referees.
  • With authors and referees, collaboratively decide which experiments the referees should attempt to replicate and how.
  • Ultimately conclude, in consultation with referees, whether the findings in the papers are sufficiently reproducible to warrant full publication.
• Authors:
  • Write the manuscript, seek feedback (e.g. via bioRxiv), and make revisions before submitting to the journal.
  • Assist referees with experimental design, reagents, and even access to personnel or specialized equipment if necessary.
• Referees:
  • Faithfully attempt to reproduce the experimental results core to the manuscript.
  • Optional: Perform any necessary additional experiments or controls to close any substantial flaws in the work.
  • Collate results.
• Readers:
  • Read the published paper and decide for themselves if the evidence supports the claims, with the confidence that the key experiments have been independently replicated by another lab.
  • Cite reproducible science.

How to Get Started

While it would be great if a journal like eLife simply piloted a peer replication pathway, I don’t think we can wait for Big Publication to initiate the shift away from traditional peer review. Maybe the quickest route would be for an organization like Review Commons to organize a trial of this new approach. They could identify some good candidates from bioRxiv and, with the authors, recruit referees to undertake the replications. Then the entire package could be shopped to journals.

I suspect that once scientists see peer replication in print, it will be hard to take seriously papers vetted only by peer review. Better science will outcompete unreproduced findings.

(Thanks Arthur Charles-Orszag for the fruitful discussions!)

experimenting with preprints

We recently uploaded a preprint to bioRxiv. The goal was to hopefully get some constructive feedback to improve the manuscript. So far, it got some tweets and even an email from a journal editor, but no comments or constructive feedback.

I notice that very few preprints on bioRxiv have any comments at all. Of course, scientists may be emailing each other privately about papers on bioRxiv, and that would be great. But I think a open process would be valuable. F1000Research, for example, has a totally open review process, posting the referee reports right with the article. I might be interested in trying that journal someday.

UPDATE: In the end, we did receive a couple emails from students who had read the preprint for their journal club. They provided some nice feedback. Super nice! We also received some helpful feedback on another preprint, and we updated the manuscript before submitting to a journal. Preprints can be super useful for pre-peer review.

readcube and deepdyve update

I just wanted to reiterate how great the ReadCube recommendations are. I imported all my PDFs and now check the recommendations every day. I often find great papers (and then later find them popping up in my RSS feeds).

Also, I wanted to let folks know that DeepDyve, the article rental site, is now allowing free 5-min rental of journal articles. Try it out!

google reader alternatives

Now that Google Reader is going the way of the dodo Google Gears, how am I going to keep up with the literature?!? I read RSS feeds of many journal table of contents, because it’s one of the best ways to keep up with all the articles out there (and see the awesome TOC art). So what am I to do?

There are many RSS readers out there (one of my favorites was Feeddler for iOS), but the real problem is syncing! Google servers took care of all the syncing when I read RSS feeds on my phone and then want to continue reading at home on my computer. The RSS readers out there are simply pretty faces on top of Google Reader’s guts.

But now those RSS programs are scrambling to build their own syncing databases. Feedly, one of the frontrunners to come out of the Google Reader retirement, claims that their project Normandy will take care of everything seamlessly. Reeder, another very popular reader, also claims that syncing will continue, probably using Feedbin. Feeddler also says they’re not going away, but with no details. After July 1, we’ll see how many of these programs actually work!

So what am I doing? I’ve tried Feedly and really like how pretty it is and easy it is to use. The real problem with Feedly is that its designed for beauty, not necessarily utility. For instance look how pretty it displays on my iPad:

But note that its hard to distinguish the journal from the authors and the abstract. And it doesn’t show the full TOC image. Feedly might be faster (you can swipe to move to the next articles), but you may not get as much full information in your brain and might miss articles that might actually interest you.

Here’s Reeder, which displays the title, journal, authors, and TOC art all differently, making it easy to quickly scan each  article:

And Feeddler:

I love that Feeddler lets me put the navigation arrow on the bottom right or left, and that it displays a lot of information in nice formatting for each entry. That way, I can quickly flip through many articles and get the full information. The major problem is that it doesn’t have a Mac or PC version, so you’ll be stuck on your phone.

I think I’ll drop Feeddler and keep demoing Reedler and Feedly until July 1 rolls around.

slate

Paul and I were interviewed for a Slate.com article about Nobel Prize predictions. More details back at my original post on the 2012 Prize.

2012 nobel prize predictions

It’s time again for my annual blog post Nobel Prize predictions. This year I’m limiting to the chemistry prizes. Of course there are many more individuals and discoveries that should be listed below and even more who deserve a Nobel Prize!

Single-Molecule Spectroscopy

Moerner [awarded in 2014], Orrit

Single-molecule imaging has matured to an important technique in biophysics. Just go to a Biophysical Society meeting and see all the talks and posters with “single molecule” in the title! Single-molecule techniques have begun to answer biological questions that would be obscured in traditional imaging. Moreover, super-resolution techniques such as PALM and STORM rely directly on detecting single molecules and the spectroscopic techniques developed in the late 80s and 90s. W.E. Moerner won the 2008 Wolf Prize in Chemistry.

Electrochemistry/Bioinorganic Electron Transfer

Bard, Gray

Al Bard won the 2008 Wolf Prize in Chemistry; Harry Gray won it in 2004.

Polymer Synthesis

Matyjaszewski, Frechet

Jean Frechet invented chemically-amplified photoresists and developed dendrimer synthesis. Kris Matyjaszewski won the 2011 Wolf Prize in Chemistry for ATRP polymerization. Of course, others were involved in both discoveries.

GPCR Structure

Kobilka, Stevens, and Palczewski

Biomolecule structures have won chemistry Nobels in the past, so I’m including G-protein coupled receptors here. A lot of buzz in the last couple years about GPCRs and Nobel. Good article here.

Update 10/10/12: Kobilka wins.

Chaperonins

Horwich, Hartl

Although these are biological molecules, they are still molecules. And many Chemistry Nobels have gone to bio-related discoveries in the last couple decades. Both won the Lasker Award in 2011.

Biomolecular Motors

Vale, Spudich, Sheetz

Another bio subject, but you really never know with the Chemistry prize. All three just won the Lasker Award this year.

(BTW, check out other predictions at ChemBark and The Curious Wavefunction and Thompson. And my prior predictions.)

(P.S. W.E. Moerner was my PhD advisor. Also, I worked in a collaboration with Kris Matyjaszewski when I was an undergrad.)

Update 9/11/12: I added chaperonins and biomolecular motors because I figure this year’s Chemistry Nobel might be more biological.

Update 10/3/12: Paul and I were interviewed for a Slate.com piece on Nobel Prize predictions. I like Paul’s section, especially about Djerassi. Anyway, here is what I said:

The line between chemistry and other fields (especially biology) is often blurred, and that’s a wonderful thing; but this fact sometimes results in a chemistry Nobel Prize being awarded for a decidedly biological discovery (like the 2009 prize for the structure of the ribosome). This may be exacerbated by the fact that the physiology or medicine prize tends to go to things directly related to health, and the chemistry prize often is used to cover the more basic biological science feats. Personally, I think it is a testament to the central position the field of chemistry holds in the Venn diagram of science.

My top prediction is for single-molecule spectroscopy. In 1989, W.E. Moerner at IBM (now at Stanford) was the first to use light (lasers) to perform measurements on single molecules. Before this, millions or trillions of molecules or more were measured together to detect an average signal. His amazingly difficult feat required ultrasensitive detection techniques, perfect samples, and temperatures just above absolute zero! A year later, Michel Orrit in France observed the fluorescent photons from a single molecule. With those early experiments, Moerner and others laid the experimental groundwork for imaging single molecules.

Single-molecule spectroscopy and imaging has become a subfield unto itself. I performed my Ph.D. research in the Moerner lab, and I know firsthand that the technique reveals events that would otherwise be hidden in averages of “bulk” measurements. Biophysics, the field of understanding how cells and biomolecules operate on a physical level, is particularly aided because rare events can have major effects in biology. (Think of a single cell mutating and then dividing into a tumor.) For example, Sunney Xie at the Pacific Northwest National Laboratory (now at Harvard) performed the early work on how individual enzymes experience multiple states, which otherwise would be averaged away in a bulk experiment. More recently, imaging single molecules has been instrumental in novel “super-resolution” techniques that reveal structures in cells at tenfold higher resolution than ever available before. Several companies (Pacific Biosciences, Helicos, Illumina, Life Technologies) have either released or are developing products that use single-molecule imaging to sequence individual strands of DNA. My prediction is bolstered by others along the same vein. In 2008, Moerner won the Wolf Prize in Chemistry, which is often considered a harbinger for the Nobel. More importantly, The Simpsons were betting on Moerner in 2010. Of course, that was Milhouse’s prediction, and maybe it’s more reasonable to go with Lisa.

My other prediction is for biomolecular motors (aka molecular motors). These are proteins in cells that move important cargo around, and on a more practical level, make muscles contract. Ron Vale (now at University of California, San Francisco) and Michael Sheetz (now at Columbia) discovered kinesin, a protein that walks along tiny tubes and pulls cargo to different parts of the cell. This is supremely important because it would take far too long (months in some cases) for diffusion alone to bring nutrients and signaling molecules to all parts of the cell. (Interestingly, kinesin was discovered from the neurons of squids because they are extraordinarily long cells!) Jim Spudich (at Stanford), Sheetz, Vale, and others have developed many important techniques for studying the actions of these tiny machines. Spudich shared this year’s Lasker Award, which many see portending a Nobel, with Vale and Sheetz.

It’s hard not to allow hope to creep into almost anything we humans do, and I have clearly failed to prevent my own desires from influencing my predictions: I would be thrilled to see either of the above discoveries—or any that I list on my blog—win a prize. But there are many, many deserving scientists who have discovered amazing things and helped millions of people. Unfortunately, only a handful of these amazing individuals will be awarded the ultimate recognition in science. So it goes.

PeerJ

This is an interesting idea. PeerJ sounds like it’s going to be an open access journal, with a cheap publication fee ($99 for a lifetime membership). I wonder if it will be selective?

I’m more excited about HHMI’s new journal eLife.

self-plagiarism and JACS

Hi all! I’m back! Well, not exactly: I won’t be posting nearly as much as I did a few years ago, but I do hope to start posting more than once a year. Sorry for my absence. There’s no real excuse except my laziness, a new postdoc position, commuting, and a new baby. I suppose those are good excuses, really. Also, I’m sorry to say, that I’ve been cheating on you, posting on another blog. We love each other, and I won’t stop, but I want to keep you Everyday Scientist readers in my live, too. I’m just not going to pay as much attention to you as I used to. You’re cool with that, right?

Anyway, I thought I’d comment on the recent blogstorm regarding Ronald Breslow’s apparently self-plagiarized JACS paper. Read the full stories here (1, 2, 3, etc.).

I feel bad for Breslow, because I like him and I respect his work and I think his paper in JACS is valuable. However, I think he should retract his paper. Sorry, but if some no-name had been caught completely copying and pasting his or her previously published paper(s) and submitting that to JACS as an ostensibly novel manuscript, that paper would be retracted when found out. If he had just copied the intro paragraph, I’d be more forgiving, but the entire document is copied (except, that is, the name of the journal)!

That said, it might be possible to save the JACS paper, but the editors would have to label the article as an Editorial or Perspective or something, and explicitly state that the article is reprinted from previous sources. I know that might not be fair, to give Breslow special treatment, but life isn’t fair. Famous scientists might get away with more than peons. And, honestly, Breslow’s paper remaining in JACS might be good for future humanity, because JACS archive will probably be more accessible than other sources. That way, we’ll be able to look up what to do when space dinosaurs visit us!

wtf?! acs fall meeting deadline is already passed?

Before the Spring meeting has even started? This is not cool.

It’s almost impossible to actually find out, but the deadline for submitting an abstract to the ACS Fall meeting in Denver has already passed. This is how I tried to find out:

First, I went to the ACS website, and clicked on the “Meetings” tab. The Fall 2011 meeting isn’t even listed there (see screenshot on the left). OK, that’s silly.

Next, I searched “deadline” from the ACS homepage and clicked on the top link, “Events & Deadlines.” That brings me to the Events & Deadlines page. Where the Denver meeting doesn’t even have a link. The Anaheim meeting’s link is live, but you can’t click on the Denver meeting. OK, maybe that means the deadline is so far away that you don’t need to worry about it. Wrong. Apparently, the Events & Deadlines page is only for past deadlines. Why have a deadlines page only for past deadlines?!? Wouldn’t future deadlines be a bit more helpful? I guess, the “Events & Deadlines” page is more a shrine to the deadlines you’ve already missed, not intended to help you meet future deadlines.

OK, let’s try going directly to the Denver meeting homepage. Not a lot of info there. But it turns out that, if you click on the symposia link, you’ll find that many of the deadlines have already passed!!! And the Spring meeting hasn’t even started yet! (There’s also this strange PDF I found somewhere on the ACS website; it list different deadlines.)

That really, really sucks. I feel like, with all the stupid emails I get from ACS every day, I’d have seen this deadline coming. I suppose it’s all my own fault: I should have been paying attention. But I figured that the deadline for the next meeting wouldn’t be before the current meeting starts. And I do blame the ACS website: I’ve been looking at the “meetings” tab for info on Denver, but it isn’t even there yet.

My suggestion: Why doesn’t ACS have one deadline for all the divisions, have it after the current meeting is finished, and actually announce that deadline on their webpage?

I am annoyed.

why is author ID taking so long?

DOI is magical. Why is it taking so long for the same thing to happen with authors? Arguably, having unique author IDs is more important and helpful than document identifiers. Yet it’s 2011 and there’s no standard way to ID an author.

Thompson has it’s ResearcherID, but it hasn’t really seem to have caught on. And it’s certainly not a open or universal standard, given it’s based off of ISI. ORCID seems to be (slowly) working on a solution to that. NIH claims that it’s working on a Pubmed Author ID project, but what’s the holdup? Hasn’t the problem of multiple authors with the same or similar name been recognized for years?

There must be some technical and economic hurdles that I don’t quite understand. DOI seemed to arrive on the scene pretty early after the internet started becoming mainstream. That was a few years ago.

chemistry should not focus on the origin of life

Several chemists (e.g. here and here) have recently suggested that the origin of life (OOL) should be the next big question the field of chemistry could tackle.

Here’s why I disagree:

• OOL research is not (directly) practical. Studying OOL won’t directly result in new technologies, products, or cures that the public can use. I prefer the Deutch and Whitesides approach. There are more pressing challenges that chemists can contribute to solving (cancer, disease, chemistry of biology, global warming, alternative energy sources, etc.). OOL comes across as an intellectual pursuit for armchair chemists.
• OOL is politically, emotionally, and religiously charged. The last thing we need is idiots trying to cut chemistry funding because their faith says something different than the science. Studying OOL is the perfect way to offend a bunch of folks and make the field of chemistry a target of religious nuts. I don’t think we should guide our research on what religious nuts want, but why kick the beehive?
• OOL is basically unanswerable. We might be able to test theories of the OOL, but we won’t be able to observe the true origins of life on this planet. Until we invent a time machine. That makes OOL research speculative and uninteresting to me. And even if we could find out, who really cares? Will that change our day-to-day life? OOL seems like more of a religious question than one of science.

Of course, some chemists should work on OOL. Just like some physicists should work on counting the number of alternate universes. But I don’t think chemistry as a whole should devote a major portion of its efforts to the “big questions” like OOL and what the universe was before the Big Bang. Chemistry is a practical science that answers questions about our everyday life. Let’s harness that power instead of trying to be as “cool” and big-question oriented as physics.

There. I hope I offended everyone who works on OOL. :)

P.S. Harry Gray and Jay Labringer have a recent editorial in Science stating that the Big Questions in chemistry are harder to see. They suggest understanding photosynthesis as one of those Questions.

what your laser pointer says about you

Red: You either don’t really care if anyone can see what you’re pointing at or you’re cheap and you use the free pointer you got from a vendor at the expo. Of course, you could be one of those considerate folks who buy very bright red pointers, because you stubbornly like what red looks like even though human eyes are not sensitive to 633 nm. That’s fine.

Green: You want your audience to see what you’re pointing at. Unless you bought a 5+ mW laser (either because you’re showing off or because you didn’t realize how sensitive the human eye is to 532 and bought the brightest laser you could find). In that case, you’re blinding your audience. If you’re going to get a 5 mW laser, get it in red. That’s classy and visible!

Blue: You’re a bad-ass. You don’t care that blue lasers are more expensive and slightly harder to see, you want the audience to know that you’re a real laser jock. (Or maybe you’re worried about leaking 1064 nm from green laser pointers.)

Purple: You’re so bad-ass you’re crazy. You don’t care that the human eye can hardly detect and can’t focus on 405 nm. You want to show that you support Blu-ray.

Yellow: You think blue lasers are soooooo 2009.

Invisible: You have a UV or IR laser pointer? Maybe a tripled or undoubled Nd:YAG? You’re nuts.

Maser pointer: I want one.