Run a hypothesis test

Status

Variants

Design and read an A/B test

Choose treatment and control units and measure the change between them.

Compare several group means

Test whether the means of several groups differ, by ANOVA and related tests.

Test fit to a distribution

Test whether values follow a given distribution, such as a fitted parametric one or Benford's law.

Compare nested models

Test whether a reduced model explains the data as well as the full model it is nested in.

Other hypothesis tests

Collections of further tests, such as rank and survival tests, shipped together.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

The goal of a hypothesis test is to establish whether certain properties of a statistical population are true by examining sample data.^[1] Usually the null hypothesis is a statement of no effect or no difference.^[1] Reach for a test when the question is a yes-or-no one about a population, such as whether a change moved a measure, and the rows in hand are a sample of that population. (Meridian's view)

The p-value is the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct.^[2] The null hypothesis is rejected when the p-value is less than or equal to the significance level of the test, denoted α.^[1] Common values for the significance level are 5% and 1%.^[1] A type I error rejects the null hypothesis when it is true, and its probability is the same as the significance level of the test.^[1]

When the data do not contradict the null hypothesis, only a weak conclusion can be made: the data provide insufficient evidence against the null hypothesis.^[1] The p-value does not give the probability that either the null hypothesis or its opposite is correct.^[1] In 2016 the American Statistical Association (ASA) stated that a p-value, or statistical significance, does not measure the size of an effect or the importance of a result.^[2] When many statistical tests are performed on the same dataset, the probability of making at least one false positive increases as the number of tests grows.^[3] The conclusion of a hypothesis test is only as solid as the sample it is based on, and the design of the experiment is critical.^[1] Decide which tests a question gets before running them, and report the size of a difference beside its p-value rather than the verdict alone. (Meridian's view)

The method

Hypothesis testing requires a statistical model of what the data would look like if chance alone were responsible for the results, called the distribution under the null hypothesis.^[1] In practice a frequentist test defines a hypothesis, selects a test statistic, derives its distribution under the null hypothesis, selects a significance level, computes the statistic from the observations and then decides.^[1] The critical region is the set of values of the test statistic for which the null hypothesis is rejected.^[1] The Neyman–Pearson decision rule rejects the null hypothesis if the observed value of the test statistic is in the critical region, and does not reject it otherwise.^[1] Common test statistics include the t-statistic of the t-test, the F-statistic of analysis of variance (ANOVA) and the chi-squared statistic of the chi-square test.^[1]

Early forms of hypothesis testing were used in the 1700s, and the first use is credited to John Arbuthnot in 1710, in analysing the human sex ratio at birth.^[1] In 1900 Karl Pearson developed the chi-squared test to determine whether a given form of frequency curve will effectively describe the samples drawn from a given population.^[1] Modern significance testing is largely the product of Karl Pearson, William Sealy Gosset and Ronald Fisher, while hypothesis testing was developed by Jerzy Neyman and Egon Pearson.^[1] The significance test Fisher popularised did not use an alternative hypothesis, so it had no concept of a type II error, a false negative.^[1] Modern hypothesis testing is an inconsistent hybrid of the Fisher and Neyman–Pearson formulations, methods and terminology developed in the early 20th century.^[1]

Compared with the alternatives

One-way ANOVA generalises the two-sample t-test when the data belong to more than two groups.^[4] In RapidMiner, the T-Test operator applies a paired t-test to every pair of performance vectors, and an additional ANOVA test is recommended because multiple paired t-tests increase the chance of a type I error.^[5]

Critics of significance testing hold that statistics should lead not to an accept-reject decision but to an estimated value with an interval estimate, an approach called estimation statistics.^[1] Bayesian inference is one proposed alternative to significance testing, and two competing hypotheses can be compared using Bayes factors.^[1] Neither Fisher's significance testing nor Neyman–Pearson hypothesis testing can give the probability that a hypothesis is true.^[1] Where the question is how large a difference is rather than whether there is one, an interval estimate answers it more directly than a test does. (Meridian's view)

Sources

1. [1]Statistical hypothesis test - Wikipediasecondary source
2. [2]p-value - Wikipediasecondary source
3. [3]Multiple comparisons problem - Wikipediasecondary source
4. [4]Student's t-test - Wikipediasecondary source
5. [5]T-Test - Rapidminer Documentationdocumentation