Theoretical Statistics Unit 6 ReviewStatistical inference

Key Concepts and Foundations

• Statistical inference involves drawing conclusions about a population based on a sample of data
• Populations are described by parameters (mean, variance) while samples are described by statistics (sample mean, sample variance)
• The central limit theorem states that the sampling distribution of the mean approaches a normal distribution as the sample size increases, regardless of the shape of the population distribution
  • This theorem is crucial for many inferential procedures
• Sufficient statistics contain all the information needed to estimate a parameter of interest
• Likelihood functions express the probability of observing the data given the parameter values
  • Maximum likelihood estimation involves finding the parameter values that maximize the likelihood function
• Bias and variance are important properties of estimators
  • Bias measures the difference between the expected value of an estimator and the true parameter value
  • Variance measures the variability of an estimator around its expected value

Probability Distributions and Their Properties

• Probability distributions describe the likelihood of different outcomes in a random experiment
• Discrete distributions (binomial, Poisson) are used for countable outcomes while continuous distributions (normal, exponential) are used for measurable outcomes
• The normal distribution is characterized by its bell-shaped curve and is determined by its mean and variance
  • Many inferential procedures assume normally distributed data or rely on the central limit theorem
• The t-distribution is similar to the normal distribution but has heavier tails and is used when the sample size is small or the population variance is unknown
• The chi-square distribution is used for inference involving variances and goodness-of-fit tests
• The F-distribution is used for comparing variances between two or more groups
• Probability density functions (PDFs) and cumulative distribution functions (CDFs) fully characterize a continuous distribution
  • PDFs give the relative likelihood of each outcome while CDFs give the probability of observing a value less than or equal to a given value

Point Estimation Techniques

• Point estimation involves using a single statistic (point estimate) to estimate a population parameter
• Method of moments estimators are found by equating sample moments (mean, variance) to population moments and solving for the parameter
• Maximum likelihood estimators (MLEs) are found by maximizing the likelihood function with respect to the parameter
  • MLEs have desirable properties such as consistency and asymptotic efficiency
• Bayesian estimation incorporates prior information about the parameter through a prior distribution
  • The prior distribution is combined with the likelihood function to obtain a posterior distribution for the parameter
• Minimum variance unbiased estimators (MVUEs) have the lowest variance among all unbiased estimators
• Sufficient statistics contain all the information in the sample relevant to estimating the parameter
  • The Rao-Blackwell theorem states that conditioning on a sufficient statistic leads to an improved estimator
• Consistency is a desirable property for point estimators, meaning that the estimator converges to the true parameter value as the sample size increases

Interval Estimation and Confidence Intervals

• Interval estimation provides a range of plausible values for a population parameter
• Confidence intervals are constructed using a point estimate and a margin of error
  • The margin of error depends on the desired confidence level (90%, 95%) and the variability of the point estimate
• The confidence level represents the proportion of intervals that would contain the true parameter value if the sampling process were repeated many times
• Wider intervals provide more confidence but less precision, while narrower intervals provide less confidence but more precision
• Confidence intervals for means are constructed using the t-distribution when the population variance is unknown and the sample size is small
• Confidence intervals for proportions are constructed using the normal distribution when the sample size is large
• Confidence intervals for variances are constructed using the chi-square distribution
• The choice of confidence level (90%, 95%, 99%) depends on the desired balance between precision and confidence
  • Higher confidence levels result in wider intervals but more certainty that the true parameter is contained in the interval

Hypothesis Testing Fundamentals

• Hypothesis testing is a procedure for determining whether a claim about a population parameter is supported by evidence from a sample
• The null hypothesis ($H_0$) represents the status quo or default position, while the alternative hypothesis ($H_a$) represents the claim being tested
• The significance level ($\alpha$) is the probability of rejecting $H_0$ when it is actually true (Type I error)
  • Common choices for $\alpha$ are 0.01, 0.05, and 0.10
• The p-value is the probability of observing a test statistic as extreme as the one observed, assuming $H_0$ is true
  • Small p-values (less than $\alpha$) provide evidence against $H_0$ and in favor of $H_a$
• The power of a test is the probability of rejecting $H_0$ when $H_a$ is actually true
  • High power is desirable and can be increased by increasing the sample size or the significance level
• One-tailed tests are used when $H_a$ specifies a direction (greater than or less than), while two-tailed tests are used when $H_a$ does not specify a direction (not equal to)
• The choice of test statistic depends on the type of data and the assumptions being made
  • Common test statistics include the z-statistic (normal distribution), t-statistic (t-distribution), and chi-square statistic (chi-square distribution)

Advanced Inference Methods

• Analysis of variance (ANOVA) is used to compare means across multiple groups
  • One-way ANOVA compares means across levels of a single factor, while two-way ANOVA compares means across levels of two factors
• The F-test is used in ANOVA to assess the overall significance of the differences among group means
• Post-hoc tests (Tukey's HSD, Bonferroni) are used to determine which specific group means differ significantly from each other
• Nonparametric tests (Wilcoxon rank-sum, Kruskal-Wallis) are used when the assumptions of parametric tests are violated
  • These tests are based on ranks rather than the actual data values
• Bootstrapping is a resampling technique used to estimate the sampling distribution of a statistic
  • Bootstrapping involves repeatedly sampling with replacement from the original sample to create many new samples
• Permutation tests are used to assess the significance of a test statistic by comparing it to a distribution generated by randomly permuting the data
• Bayesian inference updates prior beliefs about parameters based on observed data to obtain posterior distributions
  • Bayesian methods can incorporate prior information and provide probability statements about parameters

Applications in Real-world Scenarios

• Quality control uses hypothesis testing to determine whether a manufacturing process is producing items within acceptable limits
• A/B testing compares two versions of a product or website to determine which performs better
  • Hypothesis testing is used to assess whether the difference in performance is statistically significant
• Clinical trials use hypothesis testing to determine the effectiveness of new drugs or treatments compared to a placebo or standard treatment
• Polling organizations use confidence intervals to estimate the proportion of a population that supports a particular candidate or policy
• Regression analysis uses inferential methods to assess the significance of relationships between variables
  • Hypothesis tests and confidence intervals are used to determine which predictors are significantly associated with the response variable
• Time series analysis uses inferential methods to identify trends, cycles, and other patterns in data collected over time
• Machine learning algorithms use inferential methods to assess the performance of predictive models and to compare different models
  • Cross-validation and hypothesis testing are used to prevent overfitting and to select the best model

Common Pitfalls and Misconceptions

• Failing to check the assumptions of inferential procedures can lead to invalid conclusions
  • Common assumptions include normality, independence, and equal variances
• Misinterpreting p-values as the probability that $H_0$ is true or that $H_a$ is false
  • P-values only provide evidence against $H_0$, not the probability that $H_0$ or $H_a$ is true
• Confusing statistical significance with practical significance
  • A result can be statistically significant but have little practical impact, especially with large sample sizes
• Overinterpreting non-significant results as evidence of no effect
  • Failing to reject $H_0$ does not prove that $H_0$ is true, only that there is insufficient evidence to reject it
• Multiple testing can inflate the Type I error rate if not accounted for
  • Bonferroni corrections or false discovery rate control can be used to adjust for multiple comparisons
• Overfitting models to the observed data can lead to poor performance on new data
  • Regularization techniques (ridge regression, lasso) can be used to prevent overfitting
• Extrapolating conclusions beyond the scope of the data
  • Inferences should only be made about the population from which the sample was drawn, not to other populations or settings