Statistics & Data Science

Data, Questions, and Study Units

• Population and sample — the population is the full set about which an analysis seeks to learn; a sample is the observed subset. A large sample can still misrepresent its target when selection is biased.
• Unit of analysis — the entity represented by one observation in the analysis, such as a person, household, school, county, or cell. Standard errors and conclusions are wrong when dependence within higher-level units is ignored.
• Variable — a recorded attribute that varies across units. Quantitative variables encode amounts; categorical variables encode groups or labels; ordinal variables have an order without guaranteed equal spacing.
• Outcome, exposure, and covariate — an outcome is the response being explained or predicted; an exposure is a condition or intervention of interest; covariates are other measured features included for design, adjustment, precision, or description.
• Estimand — the precise population quantity a study intends to estimate. Examples include a population mean, a risk difference, an average treatment effect, or predictive error in a specified target population.
• Data-generating process — the mechanisms that produce both the underlying phenomena and the recorded data. Measurement, selection, missingness, and preprocessing are part of this process rather than nuisances outside the analysis.

• Exploratory and confirmatory analysis — exploratory work searches for patterns and helps generate hypotheses; confirmatory work tests prespecified questions under defined error controls. The same dataset can support both only when their roles are reported honestly.

• Parameter and statistic — a parameter describes a population or model; a statistic is computed from observed data. Statistics estimate parameters only relative to a defined target and sampling process.
• Unit of observation — the entity on which a measurement is recorded. It can differ from the unit of analysis, as when repeated visits are observations but the person is the independent unit.
• Data dictionary — documents each field’s meaning, type, units, allowed values, missing-value codes, derivation, and provenance so analysts do not infer semantics from column names.
• Data provenance — records where data originated and how collection, linkage, cleaning, transformation, and versioning changed them. Provenance makes errors traceable and results auditable.
• Censoring and truncation — censoring preserves partial information about an unobserved value, while truncation removes units outside an observation rule entirely. The two mechanisms require different likelihoods and interpretations.
• Long and wide data — long data store repeated measurements in multiple rows; wide data store them in multiple columns. The useful layout depends on the operation and does not change the underlying study design.

Describing Distributions

• Frequency distribution — a count or proportion for each value or interval. A histogram displays quantitative distributions; a bar chart compares discrete categories and should not imply continuity between them.
• Mean — the arithmetic average. It uses every value and is sensitive to extreme observations.
• Median — the middle ordered value, or the average of the two middle values for an even sample size. It is often more resistant than the mean to a small number of extreme observations.
• Mode — the most frequent value or category. A distribution can have one mode, several modes, or no uniquely most frequent value.
• Variance and standard deviation — variance averages squared deviations from the mean under a population or sample convention; standard deviation is its square root and returns to the variable’s original units.
• Interquartile range — the difference between the 75th and 25th percentiles. It summarizes the middle half of a distribution and is less sensitive to tails than the full range.
• Skewness — asymmetry in a distribution. Right-skewed data have a longer upper tail, often pulling the mean above the median; left-skew reverses that pattern.
• Outlier — an observation far from most others under a stated rule or model. Outliers can reflect error, rare valid cases, mixtures of populations, or model failure and should be investigated rather than automatically deleted.

• Standardization — subtracting a reference mean and dividing by a reference standard deviation produces a z-score. Standardization changes scale, not the shape or information content of the distribution.

• Quantile and percentile — a quantile is a value below which a specified proportion of observations falls; a percentile states that proportion on a 0-to-100 scale. Sample conventions can yield slightly different endpoints.
• Range — maximum minus minimum. It is easy to interpret but depends entirely on two observations and generally grows as sample size increases.
• Coefficient of variation — standard deviation divided by the mean, giving relative dispersion for positive ratio-scale measurements. It is unstable near a zero mean and inappropriate for arbitrary-origin scales.
• Covariance — measures how two variables vary together in their original units. Its sign indicates direction, while its magnitude changes when either variable is rescaled.
• Correlation — standardized covariance bounded between minus one and one for Pearson correlation. It measures linear association, can hide nonlinear structure, and does not establish causation.
• Box plot — displays a median, quartiles, whiskers, and flagged values under a chosen convention. It compresses a distribution and can conceal multimodality or small sample size.

Probability Foundations

• Sample space and event — the sample space contains possible outcomes; an event is a subset of those outcomes. A probability measure assigns values between zero and one while obeying normalization and additivity.
• Conditional probability — P(A given B) restricts attention to cases where B occurs. Dependence means P(A given B) differs from P(A); it does not by itself identify a causal relationship.
• Independence — events A and B are independent when P(A and B) equals P(A)P(B). Pairwise independence does not always imply mutual independence among three or more events.
• Bayes’ theorem — updates a probability using evidence: posterior odds equal prior odds multiplied by a likelihood ratio. The result depends on both the diagnostic or evidentiary signal and the prior prevalence.
• Random variable — a numerical function of a random outcome. Discrete variables take countable values; continuous variables are modeled over intervals and assign probability through areas under a density.
• Expected value — a probability-weighted average over possible values. It describes a long-run or model-based center and need not be a value the variable can actually take.
• Law of large numbers — under appropriate assumptions, a sample average converges toward its expectation as the number of observations grows. It does not say short sequences must resemble the theoretical distribution.

• Central limit theorem — under common regularity conditions, a standardized sum or mean approaches a normal distribution as sample size increases. The approximation can be poor for small samples, extreme skew, heavy tails, or dependence.

• Complement rule — the probability that event A does not occur is one minus the probability that A occurs. This often simplifies calculations involving at least one event.
• Union and intersection — a union contains outcomes in either event; an intersection contains outcomes shared by both. Addition requires subtracting the overlap unless the events are mutually exclusive.
• Law of total probability — calculates an event’s probability by averaging its conditional probabilities over a mutually exclusive and exhaustive partition.
• Likelihood — treats observed data as fixed and compares how strongly different parameter values or models would support them. It is not a probability distribution over parameters unless combined with additional assumptions.
• Exchangeability — a joint distribution is unchanged by specified permutations of unit labels. It can justify treating observations symmetrically but is stronger than having identical marginal distributions.
• Conditional independence — two variables may become independent after conditioning on a third. Conditioning can also create dependence, so the relevant causal or probabilistic structure must be stated.

Common Probability Distributions

• Bernoulli distribution — models one binary trial with success probability p. Its expectation is p and its variance is p(1-p).
• Binomial distribution — counts successes across a fixed number of independent Bernoulli trials with a common success probability. Overdispersion signals that these assumptions may not hold.
• Poisson distribution — models event counts under a constant-rate independent-event process, with equal mean and variance. Negative-binomial models can accommodate extra variation.
• Normal distribution — a symmetric bell-shaped family determined by mean and variance. Normality is a model assumption or approximation, not a universal property of observed data.
• t distribution — a symmetric, heavier-tailed family used when a normal mean is standardized using an estimated standard deviation. Its degrees of freedom determine tail thickness.
• Chi-square distribution — the distribution of a sum of squared independent standard normal variables. It appears in variance inference, contingency-table tests, and likelihood approximations.

• Exponential and survival distributions — the exponential distribution models a constant event hazard and is memoryless. Real survival processes often require hazards that change with time or covariates.

• Uniform distribution — assigns equal density across a continuous interval, or equal probability across a finite set. Equal density does not mean every exact continuous value has positive probability.
• Geometric distribution — counts trials until the first success under independent Bernoulli trials with constant success probability. Like the exponential distribution, it has a memoryless property.
• Negative-binomial distribution — models counts with variance exceeding the mean or, under another parameterization, trials until a fixed number of successes. The chosen parameterization must be reported.
• Hypergeometric distribution — counts successes when sampling without replacement from a finite population, making draws dependent rather than independent.
• Beta distribution — a flexible family on zero to one, often used for probabilities and proportions or as a prior for a Bernoulli probability.
• Gamma distribution — a positive continuous family used for waiting times, rates, and skewed quantities. The exponential distribution is a special case under common parameterizations.

Sampling and Measurement

• Simple random sample — every sample of a given size has the same selection probability under the classical design. Random sampling supports population generalization; random treatment assignment supports causal comparison.
• Stratified sampling — samples within prespecified subgroups, often improving precision or ensuring representation. Survey weights must reflect selection probabilities and any later adjustments.
• Cluster sampling — samples naturally grouped units such as schools or neighborhoods. Within-cluster similarity reduces effective information relative to the same number of independent observations.
• Convenience sample — recruits units because they are accessible. Precision within the observed sample does not repair unknown selection differences from the target population.
• Measurement validity — the degree to which evidence supports the intended interpretation and use of a measure. It is not a permanent property of an instrument detached from population and context.
• Reliability — consistency of measurement across repetitions, raters, items, or occasions. Reliability is necessary for many uses but does not guarantee that the intended construct is measured.
• Measurement error — the difference between a recorded value and the target quantity. Nondifferential error can bias estimates toward or away from a null depending on the model; it does not universally attenuate effects.

• Missing data — values may be missing completely at random, conditionally at random given observed information, or through mechanisms involving unobserved values. Complete-case analysis is valid only under assumptions that must be defended.

• Sampling frame — the operational list or mechanism from which units can be selected. A perfect random draw from a flawed frame still misses uncovered members of the target population.
• Coverage error — mismatch between the sampling frame and target population through undercoverage, overcoverage, duplication, or ineligible units.
• Unit nonresponse — a sampled unit provides no usable response; item nonresponse leaves particular variables missing. Weighting can reduce bias only when response mechanisms are modeled adequately.
• Response bias — systematic difference between recorded answers and target values caused by wording, recall, social desirability, mode, interviewer behavior, or other measurement conditions.
• Survey weight — represents how many target-population units an observation stands for after accounting for selection probabilities and possible nonresponse or calibration adjustments.
• Finite population correction — reduces sampling variance when a substantial fraction of a finite population is sampled without replacement. It is irrelevant for many superpopulation or process-based targets.

Estimation and Uncertainty

• Point estimator — a rule that maps sample data to one estimate, such as a sample mean or maximum-likelihood estimate. Estimators are assessed by bias, variance, consistency, robustness, and the loss relevant to the task.
• Sampling distribution — the distribution an estimator would have across repeated samples from the assumed design or model. Standard errors summarize its spread.
• Confidence interval — a procedure that captures the fixed target parameter at its nominal rate across repeated compatible samples. After one interval is observed, the classical parameter is not assigned a probability by that procedure.
• Margin of error — a critical value multiplied by an estimated standard error in many standard intervals. It accounts for sampling variation under the design, not all forms of bias or measurement error.
• Bootstrap — approximates an estimator’s sampling behavior by repeatedly resampling observed units and recomputing it. The resampling unit must respect clustering, repeated measures, time dependence, and the original design.
• Maximum likelihood — chooses parameter values that make the observed data most probable under a specified model. Likelihood measures relative support within that model and is not the probability that a parameter is true.

• Bayesian posterior — combines a prior distribution with a likelihood to obtain a probability distribution over parameters or predictions. Posterior validity depends on the likelihood, prior, computation, and whether the model represents the problem adequately.

• Estimator bias — the difference between an estimator’s expected value and its target under a specified model or design. Low bias alone does not ensure low total error.
• Consistency — an estimator converges toward its target as information grows under stated assumptions. A consistent estimator can still perform poorly in realistic finite samples.
• Efficiency — compares estimator variability for the same target under a model or design. Greater model-based efficiency can come at the cost of fragility when assumptions fail.
• Robust estimator — limits sensitivity to departures such as outliers or distributional misspecification. Robustness is always relative to particular deviations and may trade efficiency under the idealized model.
• Credible interval — a Bayesian interval containing a stated posterior probability for a parameter. Its interpretation depends on the prior, likelihood, and computation used to form the posterior.
• Posterior predictive distribution — averages predictions over posterior parameter uncertainty to describe future or replicated data under the model. It supports prediction and model checking, not automatic causal interpretation.

Hypothesis Tests and Error Rates

• Null and alternative hypotheses — formal claims used to define a test statistic and its reference distribution. A test should be tied to a scientific question and effect scale rather than treated as an automatic ritual.
• p-value — under the null model and its assumptions, the probability of a test statistic at least as extreme as the observed one. It is not the probability that the null hypothesis is true.
• Type I error — rejection of a true null under the testing framework. The significance level controls its probability for a prespecified test when the model and procedure hold.
• Type II error and power — a Type II error fails to reject a specified false null; power is the probability of rejection under a specified alternative. Power varies with effect size, noise, sample size, design, and analysis.
• Effect size — a quantitative measure of magnitude, such as a mean difference, standardized difference, risk ratio, odds ratio, or correlation. Practical meaning depends on the scale, baseline risk, uncertainty, and context.
• Multiple testing — examining many hypotheses increases opportunities for false positive findings. Family-wise error and false-discovery-rate procedures control different targets and require a defined family of tests.
• Equivalence and noninferiority — specialized designs that test whether effects lie within prespecified margins. Failure to find a conventional difference does not establish equivalence.

• Optional stopping — repeatedly testing accumulated data and stopping when a threshold is crossed inflates ordinary fixed-sample error rates unless a sequential design accounts for the monitoring rule.

• Test statistic — a data summary chosen so unusually large, small, or extreme values count against a null model. Its reference distribution must match the design and assumptions.
• One-sided and two-sided tests — a one-sided alternative specifies direction before seeing results; a two-sided alternative permits either direction. Choosing direction afterward invalidates the advertised error rate.
• Significance threshold — a prespecified rule for rejecting a null model, often expressed as alpha. It controls a long-run testing error under assumptions rather than separating important from unimportant effects.
• Permutation test — builds a reference distribution by rearranging labels or outcomes under an exchangeability condition supplied by the design or null hypothesis.
• Family-wise error rate — probability of at least one false rejection within a defined family of tests. Bonferroni control is simple and valid under broad dependence but can be conservative.
• False discovery rate — expected proportion of false rejections among all rejections under the procedure’s conditions. It differs from the probability that any one reported finding is false.

Regression and Statistical Models

• Linear regression — models a conditional mean as a linear combination of predictors. Coefficients are conditional associations under the model and become causal effects only with a valid identification strategy.
• Residual — observed minus fitted outcome. Residual plots can reveal nonlinearity, unequal variance, dependence, and influential observations that undermine a model’s intended use.
• Logistic regression — models log-odds of a binary outcome. Exponentiated coefficients are odds ratios, which can differ substantially from risk ratios when outcomes are common.
• Generalized linear model — combines a random-outcome family with a link function and linear predictor. Poisson, logistic, and ordinary linear regression are members under different choices.
• Interaction — occurs when an association or effect differs across levels of another variable on a specified scale. Absence of multiplicative interaction does not imply absence on an additive scale.
• Nonlinearity — relationships can be represented by transformations, splines, polynomials, trees, kernels, or other flexible functions. Flexibility raises the need for regularization and out-of-sample assessment.
• Multilevel model — represents variation at nested or crossed levels, such as students within schools or repeated observations within people. Partial pooling can stabilize group estimates while retaining between-group differences.

• Time-series model — accounts for ordered dependence, trend, seasonality, and shocks. Randomly shuffling time points usually destroys the structure relevant to forecasting.

• Intercept — the model’s expected outcome when all encoded predictors equal zero. Its scientific meaning depends on coding and whether zero lies within a relevant range.
• Regression coefficient — the modeled change in a conditional outcome summary for a one-unit predictor change while other included predictors are held fixed. The outcome scale and link function determine interpretation.
• Design matrix — numerical representation of predictors, transformations, indicators, and interactions supplied to a regression model. Coding choices define the comparisons its coefficients estimate.
• Omitted-variable bias — coefficient distortion that can arise when an excluded cause of the outcome is associated with an included predictor. Adding variables indiscriminately can also create bias.
• Multicollinearity — strong linear dependence among predictors, which can inflate coefficient uncertainty and make individual effects unstable without necessarily harming prediction.
• Heteroskedasticity — outcome or residual variance changes across predictor values. It can invalidate ordinary standard errors even when the conditional-mean specification is useful.

Study Design and Causal Inference

• Randomized experiment — assigns interventions by a random mechanism, balancing measured and unmeasured baseline factors in expectation. Nonadherence, attrition, interference, and missing outcomes can still complicate interpretation.
• Observational study — observes exposures without investigator assignment. Causal interpretation requires assumptions about exchangeability, positivity, consistency, measurement, and the modeled data structure.
• Confounding — mixing of an exposure-outcome contrast with effects of other causes of the outcome that differ across exposure groups. A variable can be a confounder, mediator, collider, or none depending on the causal question.
• Directed acyclic graph — represents assumed causal relationships and helps identify adjustment sets. It makes assumptions visible but cannot prove that the graph is correct.
• Selection bias — conditioning on study entry, retention, measurement, or analysis can create associations that differ from the target population relationship.
• Difference in differences — compares outcome changes across exposed and comparison groups. Causal interpretation typically relies on a parallel-trends assumption and absence of differential concurrent shocks.
• Instrumental variable — uses a variable that shifts exposure, affects the outcome only through that exposure, and is independent of relevant unmeasured causes under the stated design. These assumptions are strong and partly untestable.
• Regression discontinuity — exploits treatment assignment around a threshold. It estimates a local effect near the cutoff when units cannot precisely manipulate assignment and potential outcomes vary smoothly there.

• Target trial emulation — specifies the eligibility criteria, treatment strategies, assignment procedure, follow-up, outcomes, causal contrast, and analysis that an observational study seeks to emulate.

• Potential outcomes — define causal effects by comparing outcomes a unit would have under different interventions. Because only one is observed for each unit at a time, identification requires design and assumptions.
• Average treatment effect — the population mean difference between potential outcomes under two specified treatment strategies. The population, treatment versions, outcome, and time horizon are part of the estimand.
• Conditional exchangeability — after adjustment for a sufficient set of pre-exposure variables, treatment groups have comparable potential-outcome distributions. In observational studies this is an untestable causal assumption.
• Positivity — every covariate pattern relevant to the target comparison has a nonzero probability of receiving each treatment strategy. Sparse practical overlap can make estimates unstable before exact violations occur.
• Causal consistency — the observed outcome under the treatment actually received equals the corresponding potential outcome, requiring interventions and their relevant versions to be defined well enough for the comparison.
• Intention-to-treat effect — compares groups according to randomized assignment regardless of subsequent adherence. It preserves the randomized contrast and estimates the effect of assignment, not necessarily treatment received.

Prediction and Machine Learning

• Training, validation, and test data — training fits model parameters, validation informs model choice, and an untouched test set estimates final performance. Repeatedly consulting the test set turns it into another validation set.
• Overfitting — learning noise or sample-specific structure that fails to generalize. It is controlled through design, regularization, limited model complexity, adequate sample size, and honest validation.
• Cross-validation — rotates held-out subsets to estimate predictive performance. Splits must respect patients, families, sites, batches, or time so related observations do not leak across folds.
• Regularization — penalizes model complexity. Ridge regression shrinks coefficients; lasso can set some to zero; elastic net combines both penalties.
• Classification metrics — sensitivity, specificity, predictive values, likelihood ratios, calibration, and decision consequences answer different questions. Accuracy alone can be misleading under class imbalance.
• Calibration — agreement between predicted probabilities and observed outcome frequencies in a target population. A model can discriminate well but be poorly calibrated.
• External validation — evaluates a locked model in data meaningfully separate by place, time, or population. Random internal splits do not test many real deployment shifts.
• Dataset shift — the joint distribution of predictors and outcomes can change across environments or time. Performance monitoring must distinguish sampling noise from clinically or operationally important drift.

• Algorithmic fairness — fairness criteria formalize different ethical goals and can be mutually incompatible. Group metrics require contextual judgment about harms, access, measurement, and decision thresholds.

• Loss function — assigns a cost to prediction errors and defines what model fitting or evaluation optimizes. Squared, absolute, logarithmic, and decision-specific losses value errors differently.
• Discrimination — ability to rank or separate units with different outcomes. It does not show whether predicted probabilities are accurate or decisions improve outcomes.
• Decision threshold — converts a continuous score or probability into an action. The appropriate threshold depends on consequences, available interventions, capacity, and the target population.
• Confusion matrix — cross-tabulates predicted and observed classes into true positives, false positives, true negatives, and false negatives at a specified threshold.
• Area under the ROC curve — probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case under the evaluated population. It averages across thresholds that may be irrelevant.
• Brier score — mean squared difference between predicted probabilities and binary outcomes. It reflects both calibration and resolution and must be compared within a defined prediction setting.

Reproducibility and Interpretation

• Reproducible analysis — provides data provenance, code, software versions, parameters, and deterministic or seeded steps sufficient to regenerate results where sharing rules permit.
• Researcher degrees of freedom — flexible choices in outcomes, exclusions, transformations, subgroups, and models can produce unstable findings. Preregistration and multiverse or sensitivity analyses make this flexibility visible.
• Data leakage — information unavailable at the intended prediction time enters training or evaluation. Leakage can arise through preprocessing before splitting, future variables, duplicated units, or target-derived features.
• Sensitivity analysis — tests how conclusions change under plausible alternative assumptions, specifications, missing-data mechanisms, or unmeasured bias.
• Model checking — compares model implications with observed patterns and the intended use. Good fit to training data is insufficient when assumptions, calibration, or transportability fail.

• Association and causation — a precise association can still be noncausal, and a causal effect can be estimated imprecisely. Claims should match the design and assumptions rather than the sophistication of the model.

• Version control — records changes to code and text with recoverable history, authorship, and branches. It supports collaboration but does not by itself capture data or software environments.
• Environment capture — records operating system, package versions, dependencies, and configuration through lockfiles, containers, or equivalent manifests so computations can be reconstructed.
• Automated test — executes a defined check on code or data behavior. Useful tests target scientific invariants, transformations, edge cases, and prior failures rather than merely repeating implementation details.
• Executable report — generates narrative, tables, and figures directly from analysis code and controlled inputs, reducing manual transcription and keeping reported results synchronized.

Sources and Further Study

• OpenStax, Introductory Statistics 2e
• NIST/SEMATECH e-Handbook of Statistical Methods
• Berkeley Stat 210A notes on hypothesis testing
• The Introduction to Statistical Learning resources
• Hernán and Robins, Causal Inference: What If