Coherent Hypothesis Tests for Experimental Research

Objective

"Across psychology and allied disciplines, the statistical analysis of experimental data hinges on a comprehensive set of ""frequentist"" techniques that culminate in confidence intervals and p-values. These frequentist analysis-of-variance (ANOVA) techniques form the fossilized core of what is taught, practiced, and expected. This proposal seeks to break the mould and develop a comprehensive set of alternative ""Bayesian"" techniques that are attractive, interpretable, practically feasible, and ultimately normative as a supplement to the traditional approach. This requires that the existing set of Bayesian techniques are greatly expanded, that the resulting advantages are highlighted and demonstrated on published data, and that the methodology is supported by established open-access software that is easy to use (i.e., R and JASP; jasp-stats.org). As a result, the field stands to gain a mature alternative statistical methodology that produces new insights and allows a coherent and diagnostic test of the experimental hypotheses of interest. Key Bayesian advantages include the ability to discriminate ""absence of evidence"" from ""evidence of absence"", the ability to model-average across multiple candidate models (e.g., to account for violations of assumptions), the ability to quantify and monitor evidence as the data accumulate, and the ability to accommodate prior knowledge (e.g., to encode past empirical information in a prior distribution). The robustness of the proposed procedures is assessed using a novel ""many-statisticians"" approach, and the potential for broad adoption is enhanced by the creation of tutorial materials and a continually expanding online library of annotated examples. Although the focus of the proposal is on applications in experimental psychology, the envisioned methodology will be useful for other experimental disciplines as well. "