Why We Don't Really Know What Statistical Significance Means: Implications for Educators

@article{Hubbard2006WhyWD,
  title={Why We Don't Really Know What Statistical Significance Means: Implications for Educators},
  author={Raymond Hubbard and J. Scott Armstrong},
  journal={Journal of Marketing Education},
  year={2006},
  volume={28},
  pages={114 - 120}
}
In marketing journals and market research textbooks, two concepts of statistical significance—p values and αlevels—are commonly mixed together. This is unfortunate because they each have completely different interpretations. The upshot is that many investigators are confused over the meaning of statistical significance. We explain how this confusion has arisen and make several suggestions to teachers and researchers about how to overcome it. 

Tables from this paper

‘Are You Taking the P?’ Correcting Psychology Students’ Misconceptions About Probability Values

  • Psychology
  • 2019
Statistics are an almost unavoidable component of modern psychology and probability (or p) values are ubiquitous in published work. Despite this, many students hold misconceptions about what p values

Beyond Significance Testing: Statistics Reform in the Behavioral Sciences

This accessibly written book reviews the controversy about significance testing and introduces readers to alternative methods, especially effect size estimation (at both the group and case levels) and interval estimation (confidence intervals) in comparative studies.

Comparing Groups: Statistical Tests

This chapter addresses the question, “It looks different, but is it really different?” and involves the first inferential statistical procedures: chi-square, t-tests, and analysis of variance (ANOVA).

Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing

Reporting p values from statistical significance tests is common in psychology's empirical literature. Sir Ronald Fisher saw the p value as playing a useful role in knowledge development by acting as

Significance tests as sorcery: Science is empirical—significance tests are not

Since the 1930s, many of our top methodologists have argued that significance tests are not conducive to science. Bakan (1966) believed that “everyone knows this” and that we slavishly lean on the

Theory Psychology Charles Lambdin not significance tests are − − Significance tests as sorcery : Science is empirical

Since the 1930s, many of our top methodologists have argued that significance tests are not conducive to science. Bakan (1966) believed that “everyone knows this” and that we slavishly lean on the

PERSPECTIVE - Researchers Should Make Thoughtful Assessments Instead of Null-Hypothesis Significance Tests

Instead of making NHSTs, researchers should adapt their research assessments to specific contexts and specific research goals, and then explain their rationales for selecting assessment indicators.

Null-hypothesis significance tests in behavioral and management research: We can do better

Null-hypothesis significance tests (NHST) are a very troublesome methodology that dominates the quantitative empirical research in strategy and management. Inherent limitations and inappropriate

Null hypothesis significance tests. A mix-up of two different theories: the basis for widespread confusion and numerous misinterpretations

The theoretical origins of NHST are introduced to the scientometric community, which is mostly absent from standard statistical textbooks, and some of the most prevalent problems relating to the practice are discussed and traced back to the mix-up of the two different theoretical origins.

Searching for effects in big data: Why p-values are not advised and what to use instead

As a single-value measure of first interpretation for the scanning of Big Data this article proposes statistically secured effect sizes either based on exact, mathematically sophisticated confidence intervals for effect sizes or simplified approximations.
...

References

SHOWING 1-10 OF 30 REFERENCES

The Significance of Statistical Significance Tests in Marketing Research

Classical statistical significance testing is the primary method by which marketing researchers empirically test hypotheses and draw inferences about theories. The authors discuss the interpretation

Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers

Data analysis methods in psychology still emphasize statistical significance testing, despite numerous articles demonstrating its severe deficiencies. It is now possible to use meta-analysis to show

What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes

An improved quantitative science would emphasize the use of confidence intervals (CIs), and especially CIs for effect sizes. This article reviews some definitions and issues related to developing

Confusion Over Measures of Evidence (p's) Versus Errors (α's) in Classical Statistical Testing

Confusion surrounding the reporting and interpretation of results of classical statistical tests is widespread among applied researchers, most of whom erroneously believe that such tests are

p values, hypothesis tests, and likelihood: implications for epidemiology of a neglected historical debate.

  • S. Goodman
  • Psychology
    American journal of epidemiology
  • 1993
An analysis using another method promoted by Fisher, mathematical likelihood, shows that the p value substantially overstates the evidence against the null hypothesis.

Are null results becoming an endangered species in marketing?

Editorial procedures in the social and biomedical sciences are said to promote studies that falsely reject the null hypothesis. This problem may also exist in major marketing journals. Of 692 papers

The null ritual : What you always wanted to know about significance testing but were afraid to ask

One of us once had a student who ran an experiment for his thesis. Let us call him Pogo. Pogo had an experimental group and a control group and found that the means of both groups were exactly the

Marketing Research: Methodological Foundations

This market leader has established itself as a classic through seven previous editions by presenting superior, balanced coverage of both the qualitative and quantitative aspects of marketing

Confidence Intervals for Effect Sizes

Confidence intervals for reliability coefficients can be estimated in various ways. The present article illustrates a variety of these applications. This guidelines editorial also promulgates a