Smoking on an Airplane

by Denny Borsboom

People used to smoke on airplanes. It's hard to imagine, but it's true. In less than twenty years, smoking on airplanes has grown so unacceptable that it has become difficult to see how people ever condoned it in the first place. Psychological scientists used to refuse to share their data. It's not so hard to imagine, and it's still partly true. However, my guess is that a few years from now, data-secrecy will be as unimaginable as smoking on an airplane is today. We've already come a long way. When in 2005 Jelte Wicherts, Dylan Molenaar, Judith Kats, and I asked 141 psychological scientists to send us their raw data to verify their analyses, many of them told us to get lost - even though, at the time of publishing the research, they had signed an agreement to share their data upon request. "I don't have time for this," one famous psychologist said bluntly, as if living up to a written agreement is a hobby rather than a moral responsibility. Many psychologists responded in the same way. If they responded at all, that is.

Like Diederik Stapel.

I remember that Judith Kats, the student in our group who prepared the emails asking researchers to make data available, stood in my office. She explained to me how researchers had responded to our emails. Although many researchers had refused to share data, some of our Dutch colleagues had done so in an extremely colloquial, if not downright condescending way. Judith asked me how she should respond. Should she once again inform our colleagues that they had signed an APA agreement, and that they were in violation of a moral code?

I said no.

It's one of the very few things in my scientific career that I regret. Had we pushed our colleagues to the limit, perhaps we would have been able to identify Stapel's criminal practices years earlier. As his autobiography shows, Stapel counterfeited his data in an unbelievably clumsy way, and I am convinced that we would have easily identified his data as fake. I had many reasons for saying no, which seemed legitimate at the time, but in hindsight I think my behavior was a sign of adaptation to a defective research culture. I had simply grown accustomed to the fact that, when I entered an elevator, conversations regarding statistical analyses would fall silent. I took it as a fact of life that, after we methodologists had explained students how to analyze data in a responsible way, some of our colleagues would take it upon themselves to show students how scientific data analysis really worked (today, these practices are known as p-hacking). We all lived in a scientific version of The Matrix, in which the reality of research was hidden from all - except those who had the doubtful honor of being initiated. There was the science that people reported and there was the science that people did.

In Groningen University, where Stapel used to work, he was known as The Lord of the Data, because he never let anyone near his SPSS files. He pulled results out of thin air, throwing them around as presents to his co-workers, and when anybody asked him to show the underlying data files, he simply didn't respond. Very few people saw this as problematic, because, hey, these were his data, and why should Stapel share his data with outsiders?

That was the moral order of scientific psychology. Data are private property. Nosy colleagues asking for data? Just chase them away, like you chase coyotes from your farm. That is why researchers had no problem whatsoever denying access to their data, and that is why several people saw the data-sharing request itself as unethical. "Why don't you trust us?," I recall one researcher saying in a suspicious tone of voice.

It is unbelievable how quickly things have changed.

In the wake of the Stapel case, the community of psychological scientists committed to openness, data-sharing, and methodological transparency quickly reached a critical mass. The Open Science Framework allows researchers to archive all of their research materials, including stimuli, analysis code, and data, to make them public by simply pressing a button. The new Journal of Open Psychology Data offers an outlet specifically designed to publish datasets, thereby giving these the status of a publication. PsychDisclosure.org asks researchers to document decisions regarding, e.g., sample size determination and variable selection, that were left unmentioned in publications; most researchers provide this information without hesitation - some actually do so voluntarily. The journal Psychological Science will likely implement requirements for this type information in the submission process. Data-archiving possibilities are growing like crazy. Major funding institutes require data-archiving or are preparing regulations that do. In the Reproducibility Project, hundreds of studies are being replicated in a concerted effort. As a major symbol of these developments, we now have the Center for Open Science, which facilitates the massive grassroots effort to open up the scientific regime in psychology.

If you had told me that any of this would happen back in 2005, I would have laughed you away, just as I would have laughed you away in 1990, had you told me that the future would involve such bizarre elements as smoke-free airplanes.

The moral order of research in psychology has changed. It has changed for the better, and I hope it has changed for good.