Glossary

17.1 Binary logistic regression

2 min read•august 9, 2024

Binary logistic regression is a key statistical method for predicting outcomes with two possible values. It's used in many fields, from medicine to marketing, to figure out how different factors influence the chances of something happening.

This part of the chapter digs into the math behind logistic regression, how to fit and evaluate models, and how to use them for making predictions. You'll learn about odds ratios, , and ways to measure how well your model works.

Logistic Regression Fundamentals

Understanding the Logistic Function and Odds

Top images from around the web for Understanding the Logistic Function and Odds

  • Logistic function - Wikipedia View original

    Is this image relevant?

  • Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

    Is this image relevant?

  • Logistic function - Wikipedia View original

    Is this image relevant?

1 of 3

Top images from around the web for Understanding the Logistic Function and Odds

  • Logistic function - Wikipedia View original

    Is this image relevant?

  • Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

    Is this image relevant?

  • Logistic function - Wikipedia View original

    Is this image relevant?

1 of 3
  • Logistic function maps any real-valued number to a value between 0 and 1
  • represents the likelihood of an event occurring relative to it not occurring
  • Log-odds transforms odds to a scale from negative infinity to positive infinity
  • Sigmoid curve visualizes the logistic function, shaped like an S
  • transformation converts probabilities to log-odds, inverse of the logistic function

Mathematical Representations and Interpretations

  • Logistic function formula: f(x)=11+e−xf(x) = \frac{1}{1 + e^{-x}}
  • Odds ratio calculation: Odds=p1−p\text{Odds} = \frac{p}{1-p}, where p is the probability of success
  • Log-odds expressed as: log⁡(Odds)=log⁡(p1−p)\log(\text{Odds}) = \log(\frac{p}{1-p})
  • Sigmoid curve equation: y=11+e−(b0+b1x)y = \frac{1}{1 + e^{-(b_0 + b_1x)}}
  • Logit transformation formula: logit(p)=log⁡(p1−p)\text{logit}(p) = \log(\frac{p}{1-p})

Model Fitting and Evaluation

Estimation Techniques and Model Assessment

  • Maximum likelihood estimation finds parameter values maximizing the likelihood of observed data
  • Deviance measures the goodness of fit by comparing the model to a saturated model
  • Wald test assesses the significance of individual coefficients in the model
  • Likelihood ratio test compares nested models to determine if additional variables improve fit

Statistical Calculations and Interpretations

  • Maximum likelihood estimation uses iterative algorithms (Newton-Raphson method)
  • Deviance calculation: D=−2log⁡(likelihood of fitted modellikelihood of saturated model)D = -2 \log(\frac{\text{likelihood of fitted model}}{\text{likelihood of saturated model}})
  • Wald test statistic: W=β^SE(β^)W = \frac{\hat{\beta}}{SE(\hat{\beta})}, where β^\hat{\beta} is the estimated coefficient
  • Likelihood ratio test statistic: LR=−2log⁡(likelihood of reduced modellikelihood of full model)LR = -2 \log(\frac{\text{likelihood of reduced model}}{\text{likelihood of full model}})

Prediction and Classification

Probability Estimation and Decision Making

  • Predicted probabilities estimate the likelihood of an event occurring for given input values
  • Classification threshold determines the cutoff point for assigning binary outcomes
  • summarizes the performance of a classification model
  • plots the true positive rate against the false positive rate at various thresholds
  • AUC quantifies the overall performance of a binary classifier across all possible thresholds

Performance Metrics and Visualization

  • Predicted probabilities calculated using the logistic function: P(Y=1∣X)=11+e−(b0+b1X)P(Y=1|X) = \frac{1}{1 + e^{-(b_0 + b_1X)}}
  • Classification threshold typically set at 0.5, but can be adjusted based on specific needs
  • Confusion matrix components include true positives, true negatives, false positives, and false negatives
  • ROC curve creation involves plotting sensitivity (true positive rate) against 1-specificity (false positive rate)
  • AUC ranges from 0.5 (random classifier) to 1 (perfect classifier), with higher values indicating better performance

Key Terms to Review (16)

Binary response variable: A binary response variable is a type of categorical variable that has only two possible outcomes, often representing a 'yes' or 'no', 'success' or 'failure', or 'presence' or 'absence'. This concept is essential for modeling situations where the outcome is dichotomous and helps in analyzing relationships between predictor variables and the binary outcome. Binary response variables are crucial for understanding phenomena in various fields, including social sciences, medicine, and economics.
Car package: The car package is a collection of functions and tools in R designed to facilitate applied regression analysis, including binary logistic regression. It provides users with an intuitive way to fit models, visualize results, and conduct diagnostic tests, making it easier to interpret complex statistical outputs. The car package enhances the usability of R for statisticians and data analysts by offering robust tools specifically tailored for regression analysis.
Confusion matrix: A confusion matrix is a table used to evaluate the performance of a classification model by comparing the actual target values with the predictions made by the model. It provides a visual representation of the true positives, true negatives, false positives, and false negatives, allowing for a clear assessment of the model's accuracy, precision, recall, and F1 score. By using a confusion matrix, one can better understand how well a model classifies different categories and identify areas for improvement.
Covariate: A covariate is a variable that is possibly predictive of the outcome being studied in statistical analysis. It can help control for potential confounding factors, allowing researchers to isolate the relationship between the independent and dependent variables more clearly. In the context of models like binary logistic regression, including covariates helps improve the model's accuracy and interpretability.
Dichotomous Variable: A dichotomous variable is a type of categorical variable that has only two distinct categories or outcomes. This variable plays a crucial role in various statistical analyses, especially in binary logistic regression, where the goal is to model the relationship between one or more independent variables and a binary outcome. The simplicity of having only two categories makes dichotomous variables easy to interpret and analyze, often representing outcomes like 'yes/no', 'success/failure', or 'true/false'.
Exponentiated coefficients: Exponentiated coefficients refer to the transformation of coefficients obtained from logistic regression models, specifically in binary logistic regression, where the exponentiation of these coefficients (using the base of natural logarithm, e) converts them into odds ratios. This transformation makes it easier to interpret the relationship between predictor variables and the likelihood of the outcome occurring, as odds ratios express how a one-unit change in a predictor variable affects the odds of the outcome.
Glm(): The `glm()` function in R is used to fit generalized linear models, which extend traditional linear models to allow for response variables that follow different distributions. This function is crucial for analyzing data where the relationship between predictors and a binary or categorical outcome needs to be established, particularly through binary logistic regression and multinomial logistic regression techniques.
Hosmer-Lemeshow Test: The Hosmer-Lemeshow test is a statistical test used to assess the goodness of fit for binary logistic regression models. It evaluates how well the predicted probabilities from the model align with the observed outcomes, essentially checking if the model accurately predicts the dependent variable based on the independent variables. A significant result indicates that the model does not fit the data well, while a non-significant result suggests a good fit.
Independence of Observations: Independence of observations means that the data points collected in a study are not influenced by each other. This concept is crucial in statistical analyses, as it ensures that the results are valid and can be generalized. When observations are independent, it implies that knowing the value of one observation does not provide any information about another, making it a foundational principle in hypothesis testing and modeling.
Linearity in the logit: Linearity in the logit refers to the assumption that the relationship between the independent variables and the log-odds of the dependent binary or categorical outcome is linear. This concept is critical in both binary and multinomial logistic regression, as it ensures that the model accurately reflects how changes in predictors impact the likelihood of outcomes, represented on a log-odds scale. Violations of this assumption can lead to incorrect inferences about relationships between variables.
Logit: Logit is a function used in statistical modeling to transform probabilities into a continuous scale that can take any real number. It connects the probability of an event occurring to a linear combination of predictor variables through the logistic function. This transformation is crucial in binary logistic regression, where it helps to model the relationship between a binary outcome and one or more independent variables.
Maximum Likelihood Estimation: Maximum likelihood estimation (MLE) is a statistical method used to estimate the parameters of a probability distribution by maximizing the likelihood function. In the context of binary logistic regression, MLE helps determine the best-fitting model by finding the parameter values that make the observed data most probable, allowing for effective prediction of binary outcomes based on predictor variables.
Odds ratio: The odds ratio is a measure used in statistics to determine the odds of an event occurring in one group compared to another group. It is particularly useful in binary logistic regression, where it helps quantify the strength and direction of the association between an independent variable and a binary outcome. The odds ratio provides insights into the likelihood of an event occurring based on the presence or absence of certain characteristics.
Predictor Variable: A predictor variable is a variable that is used to predict the outcome of another variable, often referred to as the response or dependent variable. In statistical modeling, especially in binary logistic regression, predictor variables are used to determine the likelihood of an event occurring based on certain factors. These variables can be continuous, categorical, or ordinal and play a crucial role in the analysis by influencing the probability of different outcomes.
Pseudo r-squared: Pseudo r-squared is a statistical measure used to evaluate the goodness of fit of a logistic regression model, particularly in binary outcomes. Unlike the traditional r-squared in linear regression, which indicates the proportion of variance explained by the model, pseudo r-squared values provide an alternative means to assess how well a model predicts binary responses. They help in comparing models, providing insights into model performance, and interpreting the effectiveness of independent variables in explaining the dependent variable.
Roc curve: The ROC curve, or Receiver Operating Characteristic curve, is a graphical representation used to evaluate the performance of a binary classification model. It plots the true positive rate against the false positive rate at various threshold settings, allowing for a visual assessment of the trade-offs between sensitivity and specificity as the decision threshold changes.
© 2024 Fiveable Inc. All rights reserved.
AP® and SAT® are trademarks registered by the College Board, which is not affiliated with, and does not endorse this website.
Back
Practice QuizGlossary
Practice QuizGlossary
Next guide