Advantages of certainty and uncertainty

A research synthesis typically is not an endpoint in the investigation of a topic. Rarely does a synthesis offer a definitive answer to the theoretical or empirical question that inspired the investigation. Instead, most research syntheses serve as way stations along a sometimes winding research path. The goal is to describe the status of a research literature by highlighting what is unknown as well as what is known. It is the unknowns that are especially likely to suggest useful directions for new research. The purpose of this chapter is to delineate the possible relations between research syntheses, theory development, and future empirical research. By indicating the weaknesses as well as the strengths of existing research, a synthesis can channel thinking about research and theory in directions that would improve the next generation of empirical evidence and theoretical development. In general, scientists' belief about what the next steps are for empirical research and theorizing following a synthesis depends on the level of certainty that they accord its findings (see Cooper and Rosenthal 1980). In this context, certainty refers to the confidence with which the scientific community accepts empirical associations between variables and the theoretical explanations for these associations. As we explain in this chapter, the certainty accorded to findings is influenced by the perceived validity of the research. In a meta-analysis, invalidity can arise at the level of the underlying studies as well as at the meta-analytic, aggregate level. Threats to validity, as William Shadish, Thomas Cook, and Donald Campbell defined it, reduce scientists' certainty about the empirical relations and theoretical explanations considered in a meta-analysis, and this uncertainty in turn stimulates additional research (2002). Empirical relations and theoretical explanations in research syntheses range from those that the scientific community can accept with high certainty to those it can hold with little certainty only. Highly certain conclusions suggest that additional research of the kind evaluated is not required to substantiate the focal effect or validate the theoretical explanation, whereas less certain conclusions suggest a need for additional research. Our proposal to evaluate the uncertainty that remains in empirical relations and in interpretations of those relations challenges more standard ways of valuing research findings. Generally, evaluation of research, be it by journal editors or other interested scientists, favors valid empirical findings and well-substantiated theories. In this conventional approach, science progresses through the cumulation of well-supported findings and theories. Although it is standard practice for researchers to call for additional investigation at the end of a written report, such requests are often treated as rhetorical devices that have limited meaning in themselves. In our experience, editors' publication decisions and manuscript reviewers' evaluations are rarely influenced by such indicators of sources of uncertainty that could be addressed in future investigation. Following scientific convention, research syntheses would be evaluated favorably to the extent that they produce findings and theoretical statements that appear to contribute definitively to the canon of scientific knowledge. In striving to meet such criteria, the authors of syntheses might focus on what is known in preference to what is unknown. In contrast, we believe that, by highlighting points of invalidity in a research literature, research syntheses offer an alternative, generative route for scientific progress. This generative route does not produce valid findings and theoretical statements but instead identifies points of uncertainty and thus promising avenues for subsequent research and theorizing that can reduce the uncertainty. When giving weight to the generative contribution of a synthesis, journal editors and manuscript reviewers would consider how well a synthesis frames questions. By this metric, research syntheses can contribute significantly to scientific progress even when the empirical findings or theoretical understanding of them can be accorded only limited certainty. Certainty is not attached to a synthesis as a whole but rather to specific claims, such as generalizations, which vary in their truth value. This chapter will help readers identify where uncertainty is produced in synthesis findings and how it can be addressed in future empirical research and theorizing. We evaluate two goals for research syntheses and consider the guidance that they provide to subsequent research and theory. One is to integrate studies that examined a relation between variables in order to establish the size and direction of the relation. Another is to identify conditions that modify the size of the relation between two variables or processes that mediate the relation of interest. Meta-analyses that address the first goal are designed to aggregate results pertaining to relations between variables so as to assess the presence and magnitude of those relations in a group of studies. Hence these investigations include a mean effect size and confidence interval and a test of the null hypotheses, and consider whether the relation might be artifactual. In these analyses, the central focus is on establishing the direction and magnitude of a relationship. This approach is deductive to the extent that the examined relation is predicted by a theoretical account. It is inductive to the extent that syntheses are designed to establish an empirical fact as, for example, in some meta-analytic investigations of sex differences and of social interventions and medical and psychological treatments. As we explain, uncertainty in such syntheses could arise in the empirical relation itself, as occurs when the outcomes of available studies are inconclusive about the existence of a relationship, as well as in ambiguous explanations of the relation of interest. Meta-analyses that address the second goal use characteristics of studies to identify moderators or mediators of an empirical relation. Moderating variables account for variability in the size and direction of a relation between two variables, whereas mediators intervene between a predictor and outcome variable and represent a causal mechanism through which the relation occurs (see Kenny, Kashy, and Bolger 1998). Like investigations of main effects, the investigation of moderators and mediators is deductive when the evaluated relations are derived from theories, and inductive when the moderators and mediators are identified by examining the findings of the included studies. Uncertainty in these meta-analyses can emerge in the examined relations and in the explanations of findings yielded by tests of moderators and mediators. We conclude with a discussion of how research syntheses use meta-analytic estimates of mean effect sizes and evaluations of mediators and moderators in order to test competing theories. Just as testing theories about psychological and social processes lends primary research its complexity and richness, meta-analyses also can be designed to examine the plausibility of one theoretical account over others. Meta-analyses with this orientation proceed by examining the overall pattern of empirical findings in a literature, including the relation of interest and potential mediators and moderators of it, with the purpose of testing competing theoretical propositions. Uncertainty in this type of meta-analysis emerges to the extent that the data evaluated cannot firmly discriminate between the theories based on their empirical support.