Replace Peer Review with “Peer Replication”

October 13, 2021 at 1:35 pm | | literature, science and the public, science community, scientific integrity

As I’ve posted before and many others have noted, there is a serious problem with lack of adequate replication in many fields of science. The current peer review process is a dreadful combination of being both very fallible and also a huge hurdle to communicating important science.

Instead of waiting for a few experts in the field to read and apply their stamp of approval to a manuscript, the real test of a paper should be the ability to reproduce its findings in the real world. (As Andy York has pointed out, the best test of a new method is not a peer reviewing your paper, but a peer actually using your technique.) But almost no published papers are subsequently replicated by independent labs, because there is very little incentive for anyone to spend time and resources testing an already published finding. That is precisely the opposite of how science should ideally operate.

Let’s Replace Traditional Peer Review with “Peer Replication”

Instead of sending out a manuscript to anonymous referees to read and review, preprints should be sent to other labs to actually replicate the findings. Once the key findings are replicated, the manuscript would be accepted and published.

(Of course, as many of us do with preprints, authors can solicit comments from colleagues and revise a manuscript based on that feedback. The difference is that editors would neither seek such feedback nor require revisions.)

Along with the original data, the results of the attempted replication would be presented, for example as a table that includes which reagents/techniques were identical. The more parameters that are different between the original experiment and the replication, the more robust the ultimate finding if the referees get similar results.

A purely hypothetical example of the findings after referees attempt to replicate. Of course in reality, my results would always have green checkmarks.

Incentives

What incentive would any professor have to volunteer their time (or their trainees’ time) to try to reproduce someone else’s experiment? Simple: credit. Traditional peer review requires a lot of time and effort to do well, but with zero reward except a warm fuzzy feeling (if that). For papers published after peer replication, the names of researchers who undertook the replication work will be included in the published paper (on a separate line). Unlike peer review, the referees will actually receive compensation for their work in the form of citations and another paper to include on their CV.

Why would authors be willing to have their precious findings put through the wringer of real-world replication? First and foremost, because most scientists value finding truth, and would love to show that their findings hold up even after rigorous testing. Secondly, the process should actually be more rewarding than traditional peer review, which puts a huge burden on the authors to perform additional experiments and defend their work against armchair reviewers. Peer replication turns the process on its head: the referees would do the work of defending the manuscript’s findings.

Feasible Experiments

There are serious impediments to actually reproducing a lot of findings that use seriously advanced scientific techniques or require long times or a lot of resources (e.g. mouse work). It will be the job of editors—in collaboration with the authors and referees—to determine the set of experiments that will be undertaken, balancing rigor and feasibility. Of course, this might leave some of the most complex experiments unreplicated, but then it would be up to the readers to decide for themselves how to judge the paper as a whole.

What if all the experiments in the paper are too complicated to replicate? Then you can submit to JOOT.

Ancillary Benefits

Peer replication transforms the adversarial process of peer review into a cooperation among colleagues to seek the truth. Another set of eyes and brains on an experiment could introduce additional controls or alternative experimental approaches that would bolster the original finding.

This approach also encourages sharing experimental procedures among labs in a manner that can foster future collaborations, inspire novel approaches, and train students and postdocs in a wider range of techniques. Too often, valuable hands-on knowledge is sequestered in individual labs; peer replication would offer an avenue to disseminate those skills.

Peer replication would reduce fraud. Often, the other authors on an ultimately retracted paper only later discover that their coworker fabricated data. It would be nearly impossible for a researcher to pass off fabricated data or manipulated images as real if other researchers actually attempt to reproduce the experimental results. 

Potential Problems

One serious problem with peer replication is the additional time it may take between submission and ultimate publication. On the other hand, it often takes many months to go through the traditional peer review process, and replicating experiments may not actually add any time in many cases. Still this could be mitigated by authors submitting segments of stories as they go. Instead of waiting until the entire manuscript is polished, authors or editors could start arranging replications while the manuscript is still in preparation. Ideally, there would even be a  journal-blind mechanism (like ReviewCommons) to arrange reproducing these piecewise findings.

Another problem is what to do when the replications fail. There would still need to be a judgement call as to whether the failed replication is essential to the manuscript and/or if the attempt at replication was adequately undertaken. Going a second round at attempting a replication may be warranted, but editors would have to be wary of just repeating until something works and then stopping. Pre-registering the replication plan could help with that. Also, including details of the failed replications in the published paper would be a must.

Finally, there would still be the problem of authors “shopping” their manuscript. If the replications fail and the manuscript is rejected, the authors could simply submit to another journal. I think the rejected papers would need to be archived in some fashion to maintain transparency and accountability. This would also allow some mechanism for the peer replicators to get credit for their efforts.

Summary of Roles:

  • Editor:
    • Screen submissions and reject manuscripts with obviously flawed science, experiments not worth replicating, essential controls missing, or seriously boring results.
    • Find appropriate referees.
    • With authors and referees, collaboratively decide which experiments the referees should attempt to replicate and how.
    • Ultimately conclude, in consultation with referees, whether the findings in the papers are sufficiently reproducible to warrant full publication.
  • Authors:
    • Write the manuscript, seek feedback (e.g. via bioRxiv), and make revisions before submitting to the journal.
    • Assist referees with experimental design, reagents, and even access to personnel or specialized equipment if necessary.
  • Referees:
    • Faithfully attempt to reproduce the experimental results core to the manuscript.
    • Optional: Perform any necessary additional experiments or controls to close any substantial flaws in the work.
    • Collate results.
  • Readers:
    • Read the published paper and decide for themselves if the evidence supports the claims, with the confidence that the key experiments have been independently replicated by another lab.
    • Cite reproducible science.

How to Get Started

While it would be great if a journal like eLife simply piloted a peer replication pathway, I don’t think we can wait for Big Publication to initiate the shift away from traditional peer review. Maybe the quickest route would be for an organization like Review Commons to organize a trial of this new approach. They could identify some good candidates from bioRxiv and, with the authors, recruit referees to undertake the replications. Then the entire package could be shopped to journals.

I suspect that once scientists see peer replication in print, it will be hard to take seriously papers vetted only by peer review. Better science will outcompete unreproduced findings.

(Thanks Arthur Charles-Orszag for the fruitful discussions!)

2014 nobel predictions

September 12, 2014 at 10:07 am | | news, nobel, science and the public

Time for 2014 Nobel Prize predictions. Actually, it’s a little early, but with Lasker Prize announcements, I just couldn’t wait. Here’s my track record:

So here are my 2014 predictions:


ChemistryNanotechnology: Alivisatos, Whitesides, Lieber

MedicineDNA/blottingSouthern, Jefferys, Burnette

PhysicsCloaking/nonlinear optics: Pendry, Harris

Peace: Ebola: Médecins Sans Frontières


Other and past predictions:

Biomolecular motors: Vale, Sheetz, Spudich, Brady

Unfolded protein response: Walter, Mori

Soft lithography and microfluidics: Whitesides, Quake

Chaperonins: Horwich, Hartl, Lindquist, Ellis

Polymers: Frechet, Matyjaszewski, Wang, Willson

Electrochemistry/bioinorganic: Bard, Gray, Lippard

Single-molecule spectroscopy: Moerner [awarded in 2014], Orrit

Solar: Grätzel, Nocera

DNA synthesis: Caruthers

Next-gen sequencing: Webb, Craighead, Klenerman, Church …

Super-resolution optical microscopy: Betzig [awarded in 2014], Hell [awarded in 2014], Zhuang, Hess

NMR and membranes: McConnell

Electron Transfer in DNA/Electrochemical DNA Damage Sensors: Barton, Giese, Schuster

Pd-catalyzed Alkyne/Alkene Coupling and Atom-Economy: Trost

Nuclear hormone receptors: Chambon, Evans, Jensen, O’Malley

Two-photon microscopy: Webb, Denk, Strickler

DNA microarrays: Brown

AIDS: Hütter

The Pill: Djerassi

T-cell receptor: Allison [awarded in 2018], Reinherz, Kappler, Marrack


Suggestions from others:

Quantum dots: Brus

Lithium-ion batteries: Goodenough, Whittingham, Yoshino

CRISPR: Doudna

Optogenetics: Deisseroth, Zemelman, Miesenböck, Isacoff


Other predictions:

Thompson

Chemistry World

Curious Wavefunction

In the Pipeline

Inside Science

Cocktail Party Physics

Dayside

block?

April 29, 2013 at 9:15 am | | science and the public

Sidewalk infographic fail.

2013-03-22 14.55.14

You think Stanford would know how to spell Felix Bloch’s name.

great chemophobia article in Slate

February 11, 2013 at 10:02 am | | pseudoscience, science and the public

I’ve argued in the past that the Precautionary Principle is logically flawed, even dangerous. A recent article on Slate gives a great job giving an example of when the Precautionary Principle goes bad. In response to a NYT article on alternative medicine, the Slate article compares the FDA-approved drugs to the alternative medicine that a mother is more comfortable giving her son. (Surprise! the alternative medicine also contains chemicals.)

The reality is this. [The NYT author] has been tricked by the language, maliciously or not, into considering switching her child from a carefully measured weekly dose of this molecule:

Chemical structure of four marvels.

To four doses a day of an unknown amount of this chemical:

Chemical structure of a drug.

Really?

I want to be absolutely clear. Neither of these chemicals is benign or nontoxic. The LD-50 (the “lethal dose” amount that kills 50 percent of mice fed the chemical) is about the same for quercetin as it is for methotrexate, roughly 150 milligrams per kilogram of body weight.

Berberine, one of the drugs found in four-marvels powder, has been documented to cause brain damage in infants. Hello? Exactly how much of this have you been giving your son?

[The NYT author’s] “better the molecule I don’t know, than the molecule I do” stance may help her sleep better, but it is ignorance nonetheless. The chemicals are still there, even when you squint your eyes closed so you can’t see them.

This is really scary to me, that parents are giving their children unknown doses of potentially dangerous drugs. This is exactly the danger of the Precautionary Principle: people seem more comfortable with unknown dangers than known and carefully quantified risks. That’s a silly approach to risk, but I think it might just be how our brains work. And knowing that, we should be careful to guard against it.

Another concern not mentioned in either the Slate or NYT article is the drug interactions when taking a prescribed medicine with unknown alternative drugs: because they aren’t tested, alternative medicines have the potential for devastating interactions. The FDA should require at least safety testing (if not efficacy) of all medicines, both modern-medicine and alternative. NIH has an alternative medicines institute, but I wonder how quickly they can test all the options out there.

I want to also add that I completely understand the NYT mother’s concern about giving her son drugs every day. And I completely agree with the mom’s effort to find diet changes that help: the body is a complicated network, and diet can have a huge effect on health. And the immune system is in some senses a black box that we’re only beginning to understand. A variety of alternative treatments and diet changes should be tried, but eating a bunch of unknown chemicals because they have prettier names is really concerning.

I really feel for the boy and his mom, and I wish there was a magic wand to take away his pain. But even if there were, we should probably ask about the side effects of the wand.

slate

October 3, 2012 at 2:39 pm | | news, nobel, science and the public, science community

Paul and I were interviewed for a Slate.com article about Nobel Prize predictions. More details back at my original post on the 2012 Prize.

the precautionary principle is flawed

October 1, 2012 at 10:28 am | | news, science and the public

I’ve always warned against the Precautionary Principle, mainly because it has a fatal flaw: no one applies the same principle to the alternatives. The Precautionary Principle assumes a product (or medicine or technology) is harmful until it is proven to be safe, instead of the other way around. This sounds nice, but the problem is that it doesn’t take into account the dangers of the alternative products (or medicines or technologies). That is, at least how most consumers apply the principle.

I warned against this when the BPA kerfuffle emerged. Many people started to get concerned about bisphenol A, which is a monomer for polycarbonate used in many plastic bottles. Some BPA can leach from the plastic into food or liquids, and there has been some evidence that it may mimic hormones in the human body and may have negative health effects especially in children. So everyone started banning BPA bottles and switching to other materials. The main alternative is “BPA-free” plastics. When this happened, I asked, “But what are those plastics made of??”

Basically, everyone switched over from a known product (polycarbonate) that might have some deleterious effects, to a proprietary polymer (Eastman’s Tritan) that we knew nothing about. And everyone felt safe.

But what if Tritan is a thousand times more dangerous? What if the glass bottles that some people switched to leaches lead (although I doubt many parents are giving their kids crystal to drink out of)? What if those steel water bottles put chromium into your water? (The aluminum ones like Sigg are coated with a plastic, anyway.) It doesn’t really make tons of sense to throw away your old water bottles to buy brand new ones that have a new, proprietary plastic that can leach new, unknown chemicals into your water.

C&E News has a story about Eastman’s Tritan and it’s possible health dangers. We should all throw away our new water bottles and start drinking out of another unknown material so another company can make billions off of our fears. Or just start drinking directly from the faucet.

The only correct application of the Precautionary Principle is to have someone measure the safety of all the materials used to make water bottles and baby sippy cups and weigh the dangers against each other. Maybe Eastman should pay for that. ;)

(That said, I must admit that I drink out of glass, a coffee mug made in China, and a steel water bottle. Who knows what I have in my body.)

Thanks for the tip, Chemjobber.

Atoms and Molecules – A Child’s Guide to Chemistry

June 27, 2012 at 2:22 pm | | literature, science and the public, teaching

My labmate wrote a chemistry book for children … and his daughter did the illustrations. It succinctly describes atoms, orbitals, bonding, molecules, and biomolecules.

I highly recommend it.

PeerJ

June 8, 2012 at 9:39 am | | literature, science and the public, science community

This is an interesting idea. PeerJ sounds like it’s going to be an open access journal, with a cheap publication fee ($99 for a lifetime membership). I wonder if it will be selective?

I’m more excited about HHMI’s new journal eLife.

chemicals ad campaign

May 16, 2012 at 9:41 am | | science and the public, science@home

Paul has a good first draft of a chemicals ad campaign. But I was more inspired by Klaas Wynne‘s “We love … eat … live chemicals” poster:

The reason I like it is that it points out that “chemical-free” is a stupid label, and that not all chemicals are bad (at the right doses). This type of poster could be also applied to “chemical-free” shampoos, by listing what’s in natural coconut and mint oils. I also think it would be cool to draw all those chemicals (make the size of the structure correspond to the relative amount in the apple), and repeat for several “natural” and man-made products.

I think that the “We love chemicals” posters could be combined with a set of “Natural isn’t aways safe” posters. For instance, Andrea writes about an example of dangerous natural foodstuffs. And there’s always Jim Collman’s book Naturally Dangerous.

Here are my quick drafts:

I’m moderately satisfied with them.

UPDATE: MRW posted his really cool posters:

Very cool. I like them, MRW!

405 nm laser fun

May 26, 2011 at 5:39 pm | | nerd, science and the public, science@home

I bought a 10 mW (30 mW, actually, according to our lab’s power meter) 405 nm laser from Amazon. No this pointer isn’t for presentations, for reasons I have already stated. This pointer is for fun.

For instance, I’ve enjoyed shooting the beam through tonic water and seeing the fluorescence from quinine. Here’s some total internal reflection:

Any other ideas for cool “experiments”?

(Note, please be careful with this or any laser pointer. Although the purple light emanating from this pointer doesn’t look bright, it can damage your eye or skin. Even if your eyes aren’t sensitive to 405 nm, that doesn’t mean they can’t be damaged by 405 nm. This pointer is dangerous to be viewed even in diffuse reflections.)

(P.S. The sorta shitty photo credited to E.Y.L.)

UPDATE: It turns out that urine is also fluorescent:

Especially after taking a multivitamin.

chemistry should not focus on the origin of life

March 18, 2011 at 10:16 am | | science and the public, science community

Several chemists (e.g. here and here) have recently suggested that the origin of life (OOL) should be the next big question the field of chemistry could tackle.

Here’s why I disagree:

  • OOL research is not (directly) practical. Studying OOL won’t directly result in new technologies, products, or cures that the public can use. I prefer the Deutch and Whitesides approach. There are more pressing challenges that chemists can contribute to solving (cancer, disease, chemistry of biology, global warming, alternative energy sources, etc.). OOL comes across as an intellectual pursuit for armchair chemists.
  • OOL is politically, emotionally, and religiously charged. The last thing we need is idiots trying to cut chemistry funding because their faith says something different than the science. Studying OOL is the perfect way to offend a bunch of folks and make the field of chemistry a target of religious nuts. I don’t think we should guide our research on what religious nuts want, but why kick the beehive?
  • OOL is basically unanswerable. We might be able to test theories of the OOL, but we won’t be able to observe the true origins of life on this planet. Until we invent a time machine. That makes OOL research speculative and uninteresting to me. And even if we could find out, who really cares? Will that change our day-to-day life? OOL seems like more of a religious question than one of science.

Of course, some chemists should work on OOL. Just like some physicists should work on counting the number of alternate universes. But I don’t think chemistry as a whole should devote a major portion of its efforts to the “big questions” like OOL and what the universe was before the Big Bang. Chemistry is a practical science that answers questions about our everyday life. Let’s harness that power instead of trying to be as “cool” and big-question oriented as physics.

There. I hope I offended everyone who works on OOL. :)

P.S. Harry Gray and Jay Labringer have a recent editorial in Science stating that the Big Questions in chemistry are harder to see. They suggest understanding photosynthesis as one of those Questions.

why gold?

December 7, 2010 at 11:49 pm | | science and the public

So why is gold valuable? NPR’s Planet Money has a fun podcast exploring the chemical reasons that humans value gold as a currency. Prof. Sanat Kumar goes through all the elements on the periodic table, and explains the reasons that each element couldn’t be a currency. For instance, lithium can be explosive when exposed to air, noble gases are really hard to keep in your pocket, silicon is too abundant, etc.

The podcast concludes that it was inevitable that humans would choose gold as currency. Cute.

ucsf vs tsa

November 22, 2010 at 1:34 pm | | news, science and the public, stupid technology

Some profs at UCSF have concerns about the radiation dose of backscatter scanners, specifically that all the energy is deposited in the skin instead of being spread throughout the entire body. So the dose is concentrated in time and volume. Basically, it sounds like TSA hasn’t done enough safety testing on these machines.

I would like to see a risk analysis of the probability of the screening causing cancer vs. the reduced threat of airline passengers dying from terrorism. The problem is that all these are very low probability events.

Anyway, this is my response to the entire fiasco: http://www.youtube.com/watch?v=wRpWnK6Rg3E

(via Austin)

caveat grumptor

October 13, 2010 at 8:05 am | | blogs, science and the public, science community

Royce Murray doesn’t like blogs.

No that’s an oversimplification of his editorial. Actually, his worry is that science blogs are more fun and easier to read than real science journalism (which, by the way, is hard to find); meanwhile, bloggers have no required credentials, no accountability, and might just be lying to everyone.

Damn straight.

UPDATE: In case it isn’t clear, Royce Murray is one of my favorite chemists and teachers. UNC is my Alma Mater, and I really appreciated his class. While most bloggers are pretty unhappy with Royce’s editorial, I wasn’t offended. I basically agree that neither the public nor scientists should be getting information from blogs without a grain of salt. Especially this blog. I’m sarcastic 83% of the time.

NRC finally releases updated school rankings

September 28, 2010 at 3:12 pm | | news, science and the public, science community

Phew. That was a long wait. Since 1995.

But the wait wasn’t really worth it. NRC released basically a gimungous table of data, and didn’t actually give departments rankings. I don’t have time to wade through all this data. Someone needs to tell me that my department(s) are better than Harvard.

To make things worse, their spreadsheet doesn’t work on MS Excel 2008 on Mac (only on 2004). Maybe I’ll update sometime if I can wade through the data. In the meantime, see if your school has posted its own analysis. (Berkeley has. It looks like Stanford is still in the process of manipulating the data to make themselves look awesome.)

UPDATE:

Here are the top chemistry programs (ranked on research activity):

And programs containing the word “biophysic”:

So there. You can go make your own tables. I find this very confusing.

UPDATE: Not surprisingly, there are some serious errors being found in the piles and piles of numbers being released by the NRC ranking. And come to think of it, since when does Stanford chemistry have 50 faculty members??? Something is very wrong…

NRC should just list schools according to the US News and World Report rankings and keep their data hidden. jk.

Next Page >

Powered by WordPress, Theme Based on "Pool" by Borja Fernandez
Entries and comments feeds. Valid XHTML and CSS.
^Top^