An analytics concept defining statistical methods used to estimate relationships, evaluate interventions, and generate forecasts. It specifies data requirements, estimation procedures, and uncertainty measures used to support decision-making. It does not prove causation without an appropriate identification strategy, data quality checks, and sensitivity analysis. It supports performance management by translating data into estimates, predictions, and quantified uncertainty. The concept is generally stable, though tooling and best practices for measurement evolve over time.

Principle

Hypothesis Testing operationalizes falsification by comparing observed evidence to the distribution of outcomes expected under the null; decision rules (critical values or p-values) balance the risk of false positives and the goal of detecting meaningful departures.

Demonstration

In pricing experiments, the null hypothesis might state that two price points yield equal conversion rates; analysts compute a test statistic (e.g., z or chi-square), compare it to the critical region for α=0.05, and either reject or fail to reject the null.

Misapplication

Treating failure to reject as proof of no effect, selectively running many tests and reporting only significant results (multiple testing/p-hacking), or interpreting p-values as the probability the null is true.

Consequence

Correct application yields structured, reproducible decisions about evidence against the null with controlled false-positive rates; it facilitates cumulative knowledge when used with clear hypotheses and pre-specified analysis plans.

Reversal

Reversal emphasizes estimation over dichotomous testing, reporting effect sizes and uncertainty without rigid reject/fail-to-reject decisions; this approach prioritizes magnitude and precision instead of binary outcomes.

Boundary

Applies when hypotheses and test statistics are well-defined and assumptions about sampling and model distributions hold; it excludes ad-hoc, data-driven hypotheses formulated after seeing data without appropriate correction.

Semantic Tension

Tension exists between Neyman-Pearson style hypothesis testing focused on long-run error rates and Fisherian significance testing that emphasizes p-values as measures of evidence, as well as between testing and model-based inference methods like Bayesian model comparison.

Synthesis

A Hypothesis Test is the structured comparison of observed data to the null model via a statistic and decision rule to decide whether evidence warrants rejecting the null while controlling Type I error.