1,500 scientists lift the lid on reproducibility

Must read

Lucille Barrett
Lucille Barretthttps://bloggingkits.org
Future teen idol. Hardcore tv lover. Social media guru. Zombie aficionado. Travel scholar. Biker, shiba-inu lover, audiophile, Mad Men fan and proud pixelpusher. Working at the junction of minimalism and elegance to answer design problems with honest solutions. I'm fueled by craft beer, hip-hop and tortilla chips.

More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature‘s survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research.

Wordpress

The data sometimes reveal contradictory attitudes towards reproducibility. Although 52% of those surveyed agree that there is a significant ‘crisis’ of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong. Most say that they still trust the published literature.

Data on how much of the scientific literature is reproducible are rare and generally bleak. From psychology1 and cancer biology2, the best-known analyses found rates of around 40% and 10%, respectively. Our survey respondents were more optimistic: 73% said that they think that at least half of the papers in their field can be trusted, with physicists and chemists generally showing the most confidence.

The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland. “At the current time, there is no consensus on what reproducibility is or should be.” But just recognizing that is a step forward, he says. “The next step may be identifying what the problem is and to get a consensus.”

Read More Articles :

Failing to reproduce results is a rite of passage, says Marcus Munafo, a biological psychologist at the University of Bristol, UK, who has a long-standing interest in scientific reproducibility. When he was a student, he says, “I tried to replicate what looked simple from the literature and wasn’t able to. Then I had a crisis of confidence, and then I learned that my experience wasn’t uncommon.”

The challenge is not to eliminate problems with reproducibility in published work. Being at the cutting edge of science means that sometimes results will not be robust, says Munafo. “We want to be discovering new things but not generating too many false leads.”

The scale of reproducibility

But sorting discoveries from false leads can be discomfiting. Although the vast majority of researchers in our survey failed to reproduce an experiment, less than 20% of respondents said they had ever been contacted by another researcher unable to reproduce their work. Our results are strikingly similar to another online survey of nearly 900 members of the American Society for Cell Biology (see go.nature.com/kbzs2b). That may be because such conversations are difficult. If experimenters reach out to the original researchers for help, they risk appearing incompetent or accusatory or revealing too much about their own projects.

A minority of respondents reported ever having tried to publish a replication study. When work does not reproduce, researchers often assume a perfectly valid (and probably boring) reason. What’s more, incentives to publish positive replications are low, and journals can be reluctant to publish negative findings. In fact, several respondents who had published a failed replication said that editors and reviewers demanded that they play down comparisons with the original study.

Nevertheless, 24% said that they had published a successful replication, and 13% had published a failed replication. Acceptance was more common than persistent rejection: only 12% reported being unable to publish successful attempts to reproduce others’ work; 10% reported being unable to publish unsuccessful attempts.

More articles

Latest article