This week witnesses the publication of the first two articles accepted under our new Replication article type at Royal Society Open Science, which our editor Chris Chambers introduced in an earlier blog post.
This week witnesses the publication of the first two articles accepted under our new Replication article type at Royal Society Open Science, which our editor Chris Chambers introduced in an earlier blog post. Now, Chris reflects on the process while authors Hans IJzerman and Stephen Politzer-Ahles discuss their motivations for submitting a Replication and their experience with the process.
Last year, Royal Society Open Science became one of the first journals to adopt what future historians may regard as one of the most pivotal initiatives in the current era of science reform: a so-called accountable replications policy. This concept, devised by psychologist Sanjay Srivastava, commits a journal to publishing any methodologically sound replication of any study that the journal previously published, regardless of whether the original study had flaws or how the results of the replication turn out.
At Royal Society Open Science we have embraced this concept and taken it one step further by not only assuming accountability for the replicability of research published in our pages but also from dozens of other major journals. Readers can see our complete journal policy.
This week sees the first completed outputs of this initiative – one article seeking to replicate a landmark finding in personality psychology, and the other asking whether musicians are immune to a classic audio-visual illusion called the McGurk effect, as a recent study claimed. In both cases, the results did not support the original claims.
As editor of this format I’m delighted to report a successful launch and the response from authors and reviewers has been enthusiastic. As I wrote at the launch, replications are typically very difficult to publish in psychology and cognitive neuroscience because of career pressures to produce positive, novel results. But this initiative is already beginning to surface the vast and important unpublished file-drawer of methodologically sound replications.
The authors of our first two published Replications explain their studies and reflect on their experiences submitting a Replication. We look forward to receiving many more such submissions in the coming months.
What was your replication study about and what did you find?
Hans IJzerman: The replication study was one of a study that can be considered a ‘classic’ in personality psychology (it was cited more than 1100 times). The original finding was that attachment styles (whether someone feels secure or not in close relationships) and social value orientation (how generous someone is in giving fictional coins to strangers) are related: people who are secure in close relationships give more virtual coins to strangers. We did not replicate that finding in a sample that was larger than the original.
Stephen Politzer-Ahles: When the sound a person hears does not match the mouth movements they see the speaker make, the person may misinterpret the sound as being a different one (for a demonstration, listen to this demo with your eyes open and again with your eyes shut). A previous study suggested that this phenomenon does not occur among individuals with music experience, but the data analysis in that study had some crucial limitations which made the results inconclusive. We decided to attempt to replicate the study, with improvements to the design. We found that the effect does occur in both musicians and non-musicians.
What led you to submit a Replication article rather than a regular research article?
HIJ: We were very interested in the original finding. It was an important finding in the literature and we are interested in attachment, so we decided to test this again.
SP-A: We considered the Replication article type the most relevant for this study, given the aims of our study. We also are in favour of anything that provides more avenues for researchers to publish replication studies, and thus were excited to support this initiative by RSOS.
How did you find the submission and review process?
HIJ: The submission and review process was a delight. What was different from this review process is that the results were blinded and the reviewers thus did not have knowledge about our results. In addition, we encountered pretty strong bias at other journals against the replication findings and did not encounter this at RSOS.
SP-A: We were very satisfied with the process. The results-blind track gave us a good opportunity to get useful feedback and review of the methodological merits of the study. For the most part, the reviews we received were constructive. In the one case where we did get comments that we did not consider constructive and that were not statistically sound, the editor was judicious evaluating them (i.e. we did not have the experience of getting rejected based on a reviewer’s wrong claims, which we and many other colleagues have experienced at other journals before).
Would you submit more Replications to Royal Society Open Science?
HIJ: Certainly and for the reasons outlined above. Next time, I would actually prefer to do a pre-registration. We already had the results and I could not do this, but a pre-registration (having the report reviewed before we have data) is superior to what we did.
SP-A: Yes – given the positive experience I had with the review process at this journal, I consider this journal a good option if I have other replication studies to publish in the future.
What would you say to other authors considering submitting a Replication to Royal Society Open Science?
HIJ: This kind of review process (results-blind or registered reports) is vital in making our science more accurate. I hope that many scientists make use of this option.
SP-A: I would strongly encourage them to give it a try. (And in fact, my looking into RSOS for this study was also because of another colleague’s suggestion.)
Do you think other journals should offer a similar Replication policy?
HIJ: Yes, no doubt about that. Personally I am in favour that all hypothesis-driven research is done in the form of Registered or Confirmatory Report, while we still leave room for Exploratory Reports (as we are developing at the journal International Review of Social Psychology).
SP-A: Yes; while I don’t expect that every journal will have the same focus or fill the same niche, I do think this article type is valuable and it will be great to see several more journals follow RSOS’ example.
