Hypothesis Testing Reconsidered

Describes several properties of hypothesis testing, with special emphasis on analyses common to studies of perception.

Gregory Francis (Author)

9781108730716, Cambridge University Press

Paperback / softback, published 23 May 2019

75 pages, 14 b/w illus. 6 tables
23 x 15.4 x 0.5 cm, 0.15 kg

Hypothesis testing is a common statistical analysis for empirical data generated by studies of perception, but its properties and limitations are widely misunderstood. This Element describes several properties of hypothesis testing, with special emphasis on analyses common to studies of perception. The author also describes the challenges and difficulties with using hypothesis testing to interpret empirical data. Many common applications of hypothesis testing inflate the intended Type I error rate. Other aspects of hypothesis tests have important implications for experimental design. Solutions are available for some of these difficulties, but many issues are difficult to deal with.

1. Introduction
2. The basics of hypothesis testing
3. Robustness of the Two-sample t-test
4. Adding data increases the Type I error rate: optional stopping
5. ANOVA can be extremely conservative
6. ANOVA handles only one type of multiple testing problem
7. Power analyses should consider all relevant tests
8. The only p-value you can plan for is zero
9. Subjects and trials do not trade off evenly
10. Replication is a poor way to control Type I error
11. Identifying improper methods through excess success
12. Preregistration may be useful but is not necessary for good science
13. Hypothesis testing is a variation of signal detection theory
14. Using signal detection theory to analyze reported results of hypothesis testing
15. Conclusions.

Subject Areas: Perception [JMRP], Physiological & neuro-psychology, biopsychology [JMM]