We believe that it is bad policy for journals like Science to publish big, bold ideas and then leave it to subfield journals to publish replications showing that those ideas aren’t so accurate after all. Subfield journals are less visible, meaning the message often fails to reach the broader public. They are also less authoritative, meaning the failed replication will have less of an impact on the field if it is not published by Science.
3 Likes
What i am wondering is whether the journal has an ethical responsibility to publish the replication study.
If newspapers publish stuff that later prove to be “not as accurate” as they initially thought, they publish retractions or clarifications. Newspapers that fail to do so on a regular basis are not longer trusted by the readers.
Same stuff happens if there are defects/major design flaws in products such as cars/mobiles etc. The manufacturer goes to the extent of recalling the product to get it fixed.
How come Science journals do not feel that level of accountability? Shoudn’t they?
If there is a big and bold new theory published in Science then there are going to be many papers published on the subject, so I don’t know if Science is ethically bound to publish each and every one of them. However, if the weight of papers is swinging heavily in one direction then I think they should publish a review paper, preferably written by one of the scientists who initially replicated the original work. Authorship on a Science/Nature paper is a huge accomplishment, so that would add some incentive as well.
Yes and no. The real gatekeepers are the scientists themselves, but it would be helpful if the highest profile journals aided in the effort. Obviously, editors don’t have future vision, so they can’t tell if the ideas in a paper are going to pan out. They shouldn’t be punished because a scientific idea didn’t work out. If journals only published ideas that were already well supported then they wouldn’t publish any papers.
3 Likes
There could be a compelling case for Science to publish a paper of a replication study which directly refutes a previous result in it, which this study claims to be. Still, I understand that a high-profile journal like Science might not have space to publish it. Ideally, each scientific sub-field should have a journal devoted to publishing replication of high-profile results.
3 Likes
I agree, but there is a lot of gray area. If the paper is a slam dunk refutation, then they should publish it. If it calls a small part of the paper into question or if the results aren’t that clear, then it’s a bit harder to judge. If Science decides to change its policy and pledge to publish studies that contradict earlier papers, then they would probably put themselves in a no-win situation where they will tick off someone for not publishing every paper that claims to be a refutation. When you’re the most popular restaurant in town you can’t have a table for everyone.
3 Likes
They do. Retractions and corrections are published all the time. There is even an online database specifically for identifying which papers have been retracted.
1 Like
Doesn’t seem to have happened in this case.
This doesn’t seem true. If the scientists can’t get replication studies published, then they are in a weaker position.
The fact that this group had to publish their story in a magazine and now look for another publisher shows how weak they are.
How can individual scientist or small groups of scientists be considered “gate keepers” when it’s the publishers who decide what is worth publishing?
Publishing a failed replication study is not stopping journals from publishing other papers. On the contrary it will enhance the journals credibility.
I see no evidence for this. The replication problems of the study in question have been identified in a range of other articles in other journals over the last few years.
What I see here is that one journal decided not to publish the paper on grounds which had nothing to do with the actual results, and everything to do with their publishing priorities. The journal encouraged the authors to publish the study.
The authors of that article don’t mention the other replication studies published before theirs, which also found fault with the original study and failed to reproduce many of the basic findings. These studies have been published in a range of other journals over the last five years. This is probably what the editor was referring to when they noted that the field has “moved on”.
When other scholars have already published studies for years explaining that study X has methodological flaws and isn’t amenable to replication, then another study which says the same thing is going to have to bring something special to the table in order to convince an editor to publish it.
Do you have any evidence that replication studies can’t get published?
Creationist organisations, on the other hand, never retract anything until it is so widely known to be wrong that they can’t get away with repeating it any more. Even then, there is never any effort to disseminate corrections, which is why sites such as this one continue to present claims that have been known for more than thirty years to not only be false but trivially so.
2 Likes
Which is why such creationist organisations dont have the same prestige such as most scientific journals.
You need to stop equating/comparing everything with creationism.
Where is your source for this? Has the journal Science ever published any of these papers which highlight the replication issues?
Sure, in journals other than there own… that’s a pretty empty statement.
Looks like they don’t fall in line with “publication priorities” … esp for bigger journals. Atleast in this case mentioned in the Op which i was referring to.
I used a standard scholarly citation index and located a number of papers in a range of journals. This is called research.
I don’t know, I haven’t checked.
It’s not an empty statement, because it contradicts directly your claim that journals are actively suppressing the publication of these studies.
So you’re extrapolating from one data point, while ignoring all the contrary evidence.
Even though I could agree a journal has some responsibility to publish replication results, that article still left me with the strange feeling that the authors are in part mad that they didn’t get their paper into science. It could be seen in part as a way to get more prestige for their own paper.
Obviously if the journal has already published other replication studies criticizing the original paper, then they do seem to have met their responsibility, and are not under obligation to keep publishing additional negative results.
I think you are reading too much into what i wrote. The point i raised was whether journals have an ethical responsibility to publish studies which falsify a paper published by said journal earlier.
Suppressing publication of studies would involve a broader attempt to prevent other journals from publishing the paper. I don’t think i mentioned anything of the sort.
The most insightful comment I’ve seen came from a commenter on the Slate piece: the authors’ mistake was that they tried to publish the failed replication study in Science. They should have gone to Nature instead. That’s someone who understands top-tier scientific publishing.
3 Likes
They are being published, just not by Science.
Those individual scientists are the editors and peer reviewers. Modern science requires a lot of specialization, so there are a limited number of scientists that can review a paper. Journals themselves are specialized, often focusing on a narrow range of topics. Those same circle of specialized scientists meet at conferences, touch bases on where the field is, and present their latest work to one another.
Science can only publish a set number of papers a month. They have to turn down way more papers than they publish. It’s an accomplishment to get published in Science and Nature because they only publish the best papers. If Science and Nature published every paper that was submitted they wouldn’t be important journals anymore.
1 Like
You raised a bit more than that. You said science journals don’t feel the same level of accountability as newspapers.
You also suggested that scientists can’t get replication studies published.
When I said that they are published, you challenged me on it.
So you obviously didn’t believe replication studies are published (though in fact they are).
No, suppressing publication of studies can involve journals refusing to publish studies sent to them, which is exactly what you said was happening.
They do have high standards for quality, but they also only publish papers that are going to make a splash, to the extent they can. Trendiness and quality both matter.
5 Likes
I think a significant issue here is the difference between an error (typo, methodology problem, etc.) and an inability to repeat. An error would warrant a retraction or statement by the same journal. But I don’t get to publish in Science just because I was unable to repeat a study in Science. Note that nobody actually found an error in the original paper. I do think this is a good example for needing to rely on more than one study to make sweeping generalizations, which happened in this case.
5 Likes