Remembering Reasons for Reform: A More Replicable and Reproducible Communication Literature Without the Rancor

Increasing awareness of the “replication crisis” has prompted discussion about replicability and reproducibility in social and behavioral science research, including in communication. As with other fields, communication has seen discussion about concerns with the interpretation of existing research. One response has been the piecemeal adoption of “open science” practices in communication to reduce selectivity in analysis, reporting, and publication of research. Calls for further adoption of such practices have, in turn, been met with criticisms and concerns about the negative consequences of their adoption. Amidst disparate perspectives regarding solutions to replicability and reproducibility issues in communication science, difficulties building consensus and caution about negative outcomes are understandable, but they also present the risk of a status quo bias that could stall the improvement of the replicability and reproducibility of communication research. The urgency of the replication crisis for communication and the cost of inaction are presented here along three exemplifying dimensions perhaps of particular importance in communication research: (a) responsibility to the public, (b) stewardship of resources, and (c) membership in a community of scholars. While debate over solutions will continue, we would do well to keep in mind that problems with replicability and reproducibility in communication research are indeed a crisis needing immediate attention.


Introduction
This commentary discusses the relevance of the "replication crisis" to the field of communication and the urgent need to address issues with replicability and reproducibility in communication scholarship.
The following brief sections overview scholarship and discussion about concerns with replicability and reproducibility across fields and in communication specifically, emphasize a few examples of implications for which the stakes are particularly high for the field of communication, and call for cooperative and collaborative conversation in the field to ensure a valid and credible research record in communication scholarship.While there are unresolved questions about how to confront these issues in communication, this commentary emphasizes that they must indeed be confronted.

The Replication Crisis in Communication and Debates About the Implementation of Open Science Practices
Replicability and reproducibility have both been described as cornerstones of science (Moonesinghe et al., 2007;Simons, 2014).As typically used in social and behavioral science, the concepts are distinct, but both closely related to the validity of findings.Reproducibility concerns the extent to which repeating the same analyses with existing study data produces the same result originally found and reported in a study, i.e., whether the original analyses' findings can be reproduced with the data.Replicability deals with whether conducting a study again will generate similar findings to the original study, i.e., whether the original study's findings are generalizable enough that they can be observed again in other research.(Nosek et al., 2022).
A third relevant concept, robustness, refers to whether analyzing a study's data with minor variations in analyses will produce consistent results, i.e., whether the study's general findings can be observed consistently regardless of analysis strategy or whether a study's results are delicate enough to only be found with analyses employing a very particular combination of variables and cases.If a study has been conducted and reported correctly and the data curated accurately, then, its findings should in principle always be reproducible; replicability of a research finding, meanwhile, may vary from the finding being broadly observed across a vast range of settings to a finding that can only be replicated in a specific context to a finding that does not appear to be replicable even in a close attempt at a "direct replication" of a study's design.
It has been no exercise in hyperbole, then, for many in the field to use "replication crisis" to describe the burgeoning awareness over the past decade or two that vast swaths of influential empirical findings in a range of scientific fields have proven difficult to replicate and reproduce.Meta-scientific research in psychology, for example, suggests that reproducibility rates are some distance below the theoretically expected 100% (Artner et al., 2021) and that replicability rates are even lower (Open Science Collaboration, 2015).Communication research, particularly in the quantitative communication science tradition, has been among the disciplines touched by the replication crisis, and in ways that extend far beyond our collective confidence in study data or the conceptual models they support.The replication crisis has not only been a crisis of confidence in our data; in many ways, it has fomented a crisis of confidence in our community of researchers.
The most prominent studies that served as harbingers of the replication crisis (e.g., Ioannidis, 2005;Klein et al., 2014;Open Science Collaboration, 2015) were focused on the shortcomings of study designs and resultant findings vis-à-vis replicability and reproducibility.Inevitably, though, discussions of biases in designs, analyses, and processes (e.g., Ioannidis, 2005) have led to necessary, but unpleasant conversations about human biases, motivated reasoning (Simonsohn et al., 2015), and flawed behaviors by researchers behind a flawed research record.Here, conversations about replicability and reproducibility issues may understandably put researchers on the defensive as terms like "p-hacking" and "questionable research practices" (John et al., 2012) enter the parlance as grim reminders that a primary culprit for the replication crisis is at best selective flexibility in analysis, reporting, and publication of data and, at worst, deliberate misrepresentation of findings (Fanelli, 2009;Simmons et al., 2011).The same is true in communication research; some trends in findings, such as distribution of significant test results, seem unlikely to have occurred without biased action by at least some of the researchers producing them (Vermeulen et al., 2015).Discussions about replicability and reproducibility can therefore feel accusatory; even if no one study or author is under scrutiny, the implication is that someone has a thumb on the proverbial scale.This tension is exacerbated by disagreements over whether and what reform is needed in scientific practice to ensure the integrity of the replicability and reproducibility "cornerstones."A range of "open science" practices have been mooted to address this concern, from tools used to identify potential concerns with existing bodies of literature to procedures intended to increase the transparency of research practices at various points in the research and publication process.Some such practices have been adopted in communication, albeit in a somewhat piecemeal fashion across researchers and publication venues, and calls for more adoption of such practices in communication research have been made (e.g., Dienlin et al., 2021).Concerns about such reform efforts have been raised, though, including claims that enthusiasm around open science practices may be exclusionary and divisive, as well as a threat to the privacy of research participants (Fox et al., 2021).Further, the implementation of popular open science practices without adequate supervision and structure may enable "openwashing" behaviors that present the appearance of more replicable and reproducible research without actual reform (Markowitz et al., 2021).

A Common Crisis: The Urgent Human Cost of Replicability and Reproducibility Issues
As the debate about open science practices indicates, it will be difficult to reach a consensus on ideal solutions, especially considering that some practices introduce different concerns in certain specific communication research domains (e.g., health communication research involving disclosure of health information).Focusing solely on disagreements about the implementation of science reform practices in these conversations, though, may be conducive to status quo bias (Samuelson & Zeckhauser, 1988).This is a real danger, as the field's history of largely ignoring calls for a more replicable body of research (e.g., Kelly et al., 1979) should tell us that a change in process is sorely needed.Thus, the debate about how to solve replicability and reproducibility issues in communication research must remain mindful not only of the pros and cons of solutions, but of the urgent and very real cost of the issues for which solutions are needed.Here are three examples of areas perhaps particularly germane to communication as reminders that replicability and reproducibility issues in the field are both urgent and destructive.

Responsibility to the Public
Much communication research is of particular interest to the public.Audiences look to communication researchers for answers about such topics as what happens to their children when they play video games and use social media, methods to resolve conflict with romantic partners and family members, the prevalence and effectiveness of political campaign tactics, how the news media portray us and our world's issues, and the promise of communication technologies for health and education.These findings reach the public, perhaps disproportionately compared to sometimes-arcane academic research in general, via press releases, news coverage, and popular media interviews.This public communication of research findings lacks sufficient detail for audiences to evaluate the details of the research, so trust is placed in us to ensure its validity.The public is listening to us.Thus, it is urgent that we ensure we are communicating the most accurate body of research we can.

Stewardship of Resources
As demand in some academic fields plateaus and contracts, communication remains a robust academic discipline even in the increasingly uncertain economies of higher education.Graduate and undergraduate enrollment trends in communication remain relatively verdant, at least in the United States, where the number of bachelor's degrees awarded in communication, journalism, and related fields has grown during this century, especially relative to other fields in social sciences and liberal arts.The number of higher education position announcements in communication in the United States has similarly increased over the same period, even accounting for a decline since the onset of the Covid-19 pandemic, keeping pace with doctoral degree awards (National Communication Association, 2023).Even as some valuable units in universities lose faculty lines or face administrative elimination, many departments, schools, and colleges of communication gain faculty or at least maintain their size.With many research faculty contracts allocating anywhere from a considerable minority to a majority of their work efforts to research in relation to teaching and service, the result is more faculty research.With communication units across the higher-education landscape often still in growth, serviced by more faculty conducting more faculty research, communication has a proportional duty to ensure that this snowballing knowledge production is leading to a body of scholarship in which we can be confident.The field's ranks and its production of research, are still growing.Thus, it is urgent that we ensure this growing body of research is as valid as it can be.

Membership in a Community of Scholars
Again, the relatively healthy demand for communication faculty in academia supports a growing number of doctoral students and encourages the presence of early-career faculty across campuses.These early-career faculty are under disproportionate pressure to have productive programs of research to obtain terminal degrees and related achievements such as habilitation, then reach career-securing benchmarks such as long-term contracts and tenure.While protections such as tenure are in part designed to allow senior scholars to explore uncertain intellectual seas in their scholarship, it is often the early-career scholars who are most likely to be working in new empirical and conceptual areas, enlightened by fresh perspectives and armed with up-to-date interdisciplinary literature from intensive graduate study at top doctoral-granting programs.Even with direct replications underutilized in communication and elsewhere, early-career researchers are likely to be working on scholarship that at least builds on the published work of others with some degree of replication effort.Therefore, it may be these "junior" scholars who suffer most in the field when prominent studies and areas of research are plagued with replicability and reproducibility issues; already toiling to generate quality research under time pressure, their efforts are stymied when they unknowingly build their careers on the shaky foundation of previous research that resists efforts at

Conclusion
While we debate solutions to issues with replicability and reproducibility in communication, we must not lose sight of the urgency of the problem.Voices have been raising concerns about replicability in communication research since before some of our leading scholars were born (e.g., Kelly et al., 1979), and these concerns remain.We may disagree on how best to address problems with replicability and reproducibility in communication research, but these cornerstones of science are essential to the continued credibility of our field among scholars and the public.Systemic problems with our research record lead to a misinformed public, a growing body of flawed studies, and mounting pressure on our most innovative and vulnerable young scholars.These are severe potential consequences for communication research and the scholars engaged in it.Thoughtful and productive debate over solutions is healthy for our field, but failing to act to ensure the consistent integrity of our research standards and published body scholarship is not an option.The path ahead may be uncertain, but the very real cost of inaction is clear.
Media and Communication• 2024 • Volume 12 • Commentary 7852 replication.Our best new scholars build on the scholarship of others.Thus, it is urgent that we ensure they have replicable research on which to build.