Logistic Regression - Practical

1 Introduction

In this practical, you will complete the following steps to build a logistic regression model:

  • Explore the data before modeling.
  • Fit and interpret logistic regression models.
  • Evaluate model performance using ROC, AUC, and confusion matrix.
  • Understand interaction effects to uncover complex relationships.

2 Load Relevant Libraries

First, ensure the following libraries are installed and loaded:

# Load required libraries
library(ggplot2)
library(caret)
Loading required package: lattice
library(pROC)
Type 'citation("pROC")' for a citation.

Attaching package: 'pROC'
The following objects are masked from 'package:stats':

    cov, smooth, var
library(ROCR)

3 Load Dataset

3.1 Task

  1. Load the following dataset.
  2. Display the first few rows.
  3. Summarise its statistical properties to understand the structure and distributions.

Dataset location:

https://www.dropbox.com/scl/fi/9f7at6cnh4otzt8a6yzis/log_reg_data.csv?rlkey=tgvdzq9xzkjr2m8khi45b08mc&dl=1

3.2 Solution

Code 
rm(list=ls())

football_data <- read.csv("https://www.dropbox.com/scl/fi/9f7at6cnh4otzt8a6yzis/log_reg_data.csv?rlkey=tgvdzq9xzkjr2m8khi45b08mc&dl=1")
head(football_data)
summary(football_data)

4 Exploratory Data Analysis (EDA)

4.1 Instructions

  1. Summarise the dataset using descriptive statistics.
  2. Visualise the relationship between each predictor and the outcome (score_outcome).

4.2 Solution

Code 
# Summary of the dataset
summary(football_data)

# Visualisation example: Minutes Played vs Score Outcome
ggplot(football_data, aes(x=score_outcome, y=minutes_played, fill=score_outcome)) +
  geom_violin(trim=FALSE, alpha=0.5) +
  geom_boxplot(width=0.1, outlier.shape=NA) +
  geom_jitter(width=0.1, alpha=0.3, color="black") +
  labs(title="Minutes Played vs Score Outcome", x="Score Outcome", y="Minutes Played") +
  theme_minimal() +
  theme(legend.position="none")

5 Fit a Logistic Regression Model

5.1 Instructions

  1. Fit a logistic regression model to predict score_outcome using all predictors. You need to convert the outcome to a numeric variable (0 = No Score, 1 = Score).
  2. Interpret the model summary.

5.2 Solution

Code 
# Convert score_outcome to numeric (0 = No Score, 1 = Score)
football_data$score_outcome_binary <- ifelse(football_data$score_outcome == "Score", 1, 0)


# Fit logistic regression model using the binary outcome
logit_model <- glm(score_outcome_binary ~ player_position + minutes_played + shots_on_target + assists + defensive_pressure + fatigue_level, 
                   data = football_data, family = binomial)

summary(logit_model)

6 A Note on Factors in Regression

6.1 Interpreting Factor Variables in Logistic Regression

In logistic regression, factor variables represent categorical data (like player_position) with two or more categories. R handles these factors by creating dummy variables (also known as indicator variables).

6.2 Key Concepts

  1. Reference Category: R automatically selects one category as the reference (usually the first level alphabetically unless specified). The coefficients for other categories are interpreted relative to this reference.

  2. Interpretation of Coefficients:

    • A positive coefficient means the category increases the log-odds of scoring compared to the reference.
    • A negative coefficient decreases the log-odds of scoring.

  3. Odds Ratios: Converting coefficients to odds ratios (exp(coef)) helps in understanding the magnitude of the effect.

6.3 Example

# Check the levels of player_position
levels(football_data$player_position)
NULL
# Fit logistic regression with player_position
logit_model <- glm(score_outcome_binary ~ player_position + minutes_played + shots_on_target + assists + defensive_pressure + fatigue_level, 
                   data = football_data, family = binomial)
summary(logit_model)

Call:
glm(formula = score_outcome_binary ~ player_position + minutes_played + 
    shots_on_target + assists + defensive_pressure + fatigue_level, 
    family = binomial, data = football_data)

Coefficients:
                          Estimate Std. Error z value Pr(>|z|)    
(Intercept)               -2.57361    1.32565  -1.941 0.052211 .  
player_positionForward     0.58691    0.48764   1.204 0.228749    
player_positionMidfielder -0.30626    0.52452  -0.584 0.559295    
minutes_played             0.03537    0.01425   2.481 0.013096 *  
shots_on_target            0.25403    0.11530   2.203 0.027582 *  
assists                    0.43043    0.23134   1.861 0.062797 .  
defensive_pressure        -0.26614    0.08671  -3.069 0.002145 ** 
fatigue_level             -0.35567    0.10357  -3.434 0.000594 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 197.36  on 199  degrees of freedom
Residual deviance: 156.09  on 192  degrees of freedom
AIC: 172.09

Number of Fisher Scoring iterations: 5
# Convert coefficients to odds ratios
exp(coef(logit_model))
              (Intercept)    player_positionForward player_positionMidfielder 
               0.07625955                1.79842784                0.73619569 
           minutes_played           shots_on_target                   assists 
               1.03600009                1.28921430                1.53792329 
       defensive_pressure             fatigue_level 
               0.76632908                0.70070270

6.4 How to Read This

  • If "Defender" is the reference:
    • A positive coefficient for "Forward" means forwards are more likely to score compared to defenders.
    • A coefficient near zero for "Midfielder" means their scoring likelihood is similar to defenders.

7 Interpret Model Coefficients

7.1 Instructions

  1. Convert the model coefficients to odds ratios.
  2. Interpret the odds ratios for each predictor.

7.2 Solution

Code 
# Convert log-odds to odds ratios
exp(coef(logit_model))

8 Model Diagnostics

8.1 Instructions

  1. Calculate predicted probabilities for each observation.
  2. Plot the distribution of predicted probabilities.
  3. Evaluate model performance using the ROC curve and AUC.

8.2 Solution

Code 
# Predicted probabilities
football_data$predicted_prob <- predict(logit_model, type = "response")

# Plot predicted probabilities
ggplot(football_data, aes(x=predicted_prob, fill=score_outcome)) +
  geom_histogram(binwidth=0.05, position="dodge") +
  labs(title="Predicted Probability Distribution", x="Predicted Probability", fill="Score Outcome")
Code 
# ROC Curve and AUC
roc_curve <- roc(football_data$score_outcome, football_data$predicted_prob)
plot(roc_curve, main="ROC Curve")
Code 
auc(roc_curve)

9 Model Evaluation Using Confusion Matrix

9.1 Instructions

  1. Create a confusion matrix based on a 0.5 probability threshold.
  2. Interpret the matrix (accuracy, sensitivity, specificity).

Note: solving this task requires careful coding to ensure the predicted values (0,1) match those in the original dataset!

9.2 Solution

Code 
# Predicted classes
football_data$score_outcome_binary <- ifelse(football_data$score_outcome == "Score", 1, 0)

# Predicted classes
predicted_class <- ifelse(football_data$predicted_prob > 0.5, 1, 0)
predicted_class <- factor(predicted_class, levels = c(0, 1))

# Convert actual outcome to factor
actual_outcome <- factor(football_data$score_outcome_binary, levels = c(0, 1))

# Confusion Matrix
confusionMatrix(predicted_class, actual_outcome)

10 Interaction Effects

10.1 Instructions

  1. Fit a logistic regression model including an interaction between player_position and shots_on_target.
  2. Interpret the interaction term.

10.2 Solution

Code 
# Logistic regression with interaction term
logit_interaction_model <- glm(score_outcome_binary ~ player_position * shots_on_target + minutes_played + assists + defensive_pressure + fatigue_level, 
                               data = football_data, family = binomial)
summary(logit_interaction_model)

11 Interpretation

What does your model tell you about the effect of the predictor variables, and their interactions, on the likelihood of scoring or not scoring?