Royal Society Open Science published its first Registered Report last week. Here, the authors discuss their research and the Registered Report publishing experience.
Last week we published what we anticipate will be the first of many Registered Reports in Royal Society Open Science, and our Editor Chris Chambers talked about the article type in an accompanying blog post. Now, Chris interviews the authors Kaylin Ratner, Felix Thoemmes, and Anthony Burrow to explore their motivations for submitting a Registered Report to Royal Society Open Science and discuss their experience with the process.
What is your article about?
Kaylin Ratner: Our article set out to directly replicate the effects of Heintzelman, Trent, and King (2013; published in Psychological Science) who demonstrated
that exposure to coherent or incoherent stimuli could significantly impact perceived meaning in life. After preregistering our paper with both Royal Society Open Science and the Open Science Framework (OSF), we collected the data and tested the effects of interest. Our findings did not successfully replicate the effects that were reported in Heintzelman et al. (2013) and our paper goes on to discuss possible reasons for this discrepancy. Furthermore, we propose that future direct and conceptual replications of this effect are needed to adjudicate between these reports in addition to helping the field understand the conditions necessary for this effect to manifest.
Why did you consider a Registered Report for your research?
KR: The goal for psychologists and researchers, as a field, is to produce good science. Without good science, we have no hope for capitalizing on our knowledge to help improve the daily lives of individuals. Given the replication crisis in psychology elucidated by Nosek and his colleagues in 2015, we had an interest in helping the field move toward more accurate understandings of phenomena especially following several conceptual replications on our part which were not able to replicate the original studies’ results. With our situation and the current zeitgeist, completing a Registered Report seemed like the right thing to do.
Felix Thoemmes: Replication and Open Science
are part of the curriculum in Quantitative Methods at Cornell. It felt natural to make the next step to actuallyconduct a pre-registered replication. The million dollar pre-registration challenge from the OSF was an additional nudge to conduct the study.
What are the benefits to you as authors in submitting a Registered Report?
KR: I felt like this process restored my confidence in the scientific method. As I begin my career, I am quickly realizing that my previous conceptions of the empirical process were quite idealized. As an emerging scholar, I am beginning to recognize that reading published literature is a bit like encountering an iceberg insofar as you only see the finished, pretty product and nothing that lurks below the water’s surface. I feel strongly that this is a cultural shift that we – as emerging scholars – need to embrace. There is no reason why we should not be more transparent with our work. Registered Reports and pre-registration more generally, I feel, will help us to move away from the positivity bias that has historically plagued the publication system. Further, I feel that Registered Reports will help us humanize the scientific process and make our findings more accessible to the public.
FT: Having a well-formulated analysis plan was a major advantage. We actually wrote the complete R code before data collection, and once data were collected, the actual analysis was straight-forward and very efficient.
Anthony Burrow: Our particular Registered Report
was a replication of a previously published effect. Thus, registering our work gave us a chance to document each of our steps in a transparent way. Regardless of what we found, readers would be able to follow each decision we made, and every step of our analysis.
What challenges did you face in preparing your Registered Report, and how did you overcome them?
KR: The greatest challenge for me was framing our paper. I did not want anyone to perceive our paper as a “witch hunt” because that interpretation could not be further from the truth. On one hand, I felt like it was important to objectively present our findings and, on the other, I felt like we had to be very cautious in our wording because it can be easy to interpret unsuccessful replications as evidence for the invalidity of the original study. More replications are necessary to ascertain true effects and I cannot stress this point enough.
FT: Navigating the pre-registration process simultaneously with OSF and the journal was sometimes challenging because we wanted to make sure that we submitted all documents to the right place at the right time. Luckily, both OSF and the journal were extremely responsive to our questions.
What would you say to other authors considering submitting Registered Reports to Royal Society Open Science?
KR: I would highly recommend this option and Royal Society Open Science in particular. It was my impression that the journal’s intentions were congruent with ours in that we both simply wanted to produce good science. The review process was extremely helpful and I was particularly excited to have the self-identified original authors of the paper we were trying to replicate review our pre-registered replication. It made me feel like we were all on the same page headed toward the same goal – there were no “sides.” I also really appreciated the extent to which the journal worked with us through this journey.
FT: Registered Reports will undoubtedly become even more important and common in the future. Royal Society Open Science provides a wonderful outlet for this kind of work.
AB: The process is simple and straightforward. With so many researchers conducting replication studies, it is an intuitive decision to preregister these investigations and enjoy all of the benefits that go along with doing so.
Would you submit more Registered Reports to Royal Society Open Science? Why?
KR: I would absolutely submit more Registered Reports to Royal Society Open Science. I was honestly surprised to learn that we were the first Registered Report to be accepted by the journal because the process was so smooth. Although I found the Registered Reports process to be more demanding upfront, I whole-heartedly believe that it was worth it. Once we had Stage 1 completed and approved, everything else felt like it fell into place and it was a welcomed change in contrast to the traditional publication process.
FT: All of us have a strong commitment to open science and pre-registration, so a future submission to Royal Society Open Science is likely. During the process of this pre-registration I have also signed the transparency in research commitment and the Peer Reviewers’ Openness Initiative, and would encourage others to do so as well.
(Editor’s note: readers may also be interested in the paper published by Royal Society Open Science on the Peer Reviewers’ Openness Initiative.)
How did you find the submission and review process for your Registered Report?
KR: As should be apparent from my responses above, I found the submission and review process for the Registered Report to be extremely well-organized and even-handed. All aspects of this process were tremendously professional and I could not have been more pleased. Again, it is hard to believe that we were the first Registered Report with this journal judging by the speed and efficiency.
You can read the full Registered Report at The effects of exposure to objective coherence on perceived meaning in life: a preregistered direct replication of Heintzelman, Trent & King (2013). Royal Society Open Science welcomes enquiries about Registered Reports, and you can submit your work online — we look forward to hearing from you!
