Telco Churn Code

The following code was written to apply and evaluate the performance of Logistic Regression and Support Vector Machine on a customer churn dataset for predicting customer churn

Importing Dependencies

Dependencies are external libraries which contain prewritten software to help us achieve a given task

import pandas as pd # For reading data in from a file
from sklearn.preprocessing import StandardScaler # For scaling/normalising numeric values in the data
from sklearn.model_selection import train_test_split # For splitting the data into train and test sets
from sklearn.linear_model import LogisticRegression # For building the Logistic Regression model
from sklearn.metrics import classification_report, roc_auc_score # For evaluating the model's performance
from sklearn.model_selection import GridSearchCV # For hyperparameter tuning
from sklearn.svm import SVC # For building the SVM model

# For Randomised Grid Search
from sklearn.model_selection import RandomizedSearchCV
import numpy as np

Reading the dataset from a file...

...and sampling some of the data

df = pd.read_excel("Telco_customer_churn.xlsx") # Read the dataset from an Excel file into a dataframe

df.head() # Sample the first few rows of the dataset

	CustomerID	Count	Country	State	City	Zip Code	Lat Long	Latitude	Longitude	Gender	...	Contract	Paperless Billing	Payment Method	Monthly Charges	Total Charges	Churn Label	Churn Value	Churn Score	CLTV	Churn Reason
0	3668-QPYBK	1	United States	California	Los Angeles	90003	33.964131, -118.272783	33.964131	-118.272783	Male	...	Month-to-month	Yes	Mailed check	53.85	108.15	Yes	1	86	3239	Competitor made better offer
1	9237-HQITU	1	United States	California	Los Angeles	90005	34.059281, -118.30742	34.059281	-118.307420	Female	...	Month-to-month	Yes	Electronic check	70.70	151.65	Yes	1	67	2701	Moved
2	9305-CDSKC	1	United States	California	Los Angeles	90006	34.048013, -118.293953	34.048013	-118.293953	Female	...	Month-to-month	Yes	Electronic check	99.65	820.5	Yes	1	86	5372	Moved
3	7892-POOKP	1	United States	California	Los Angeles	90010	34.062125, -118.315709	34.062125	-118.315709	Female	...	Month-to-month	Yes	Electronic check	104.80	3046.05	Yes	1	84	5003	Moved
4	0280-XJGEX	1	United States	California	Los Angeles	90015	34.039224, -118.266293	34.039224	-118.266293	Male	...	Month-to-month	Yes	Bank transfer (automatic)	103.70	5036.3	Yes	1	89	5340	Competitor had better devices

5 rows × 33 columns

Exploratory Data Analysis

df.info() # Explore the columns

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7043 entries, 0 to 7042
Data columns (total 33 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   CustomerID         7043 non-null   object 
 1   Count              7043 non-null   int64  
 2   Country            7043 non-null   object 
 3   State              7043 non-null   object 
 4   City               7043 non-null   object 
 5   Zip Code           7043 non-null   int64  
 6   Lat Long           7043 non-null   object 
 7   Latitude           7043 non-null   float64
 8   Longitude          7043 non-null   float64
 9   Gender             7043 non-null   object 
 10  Senior Citizen     7043 non-null   object 
 11  Partner            7043 non-null   object 
 12  Dependents         7043 non-null   object 
 13  Tenure Months      7043 non-null   int64  
 14  Phone Service      7043 non-null   object 
 15  Multiple Lines     7043 non-null   object 
 16  Internet Service   7043 non-null   object 
 17  Online Security    7043 non-null   object 
 18  Online Backup      7043 non-null   object 
 19  Device Protection  7043 non-null   object 
 20  Tech Support       7043 non-null   object 
 21  Streaming TV       7043 non-null   object 
 22  Streaming Movies   7043 non-null   object 
 23  Contract           7043 non-null   object 
 24  Paperless Billing  7043 non-null   object 
 25  Payment Method     7043 non-null   object 
 26  Monthly Charges    7043 non-null   float64
 27  Total Charges      7043 non-null   object 
 28  Churn Label        7043 non-null   object 
 29  Churn Value        7043 non-null   int64  
 30  Churn Score        7043 non-null   int64  
 31  CLTV               7043 non-null   int64  
 32  Churn Reason       1869 non-null   object 
dtypes: float64(3), int64(6), object(24)
memory usage: 1.8+ MB

# Disable scientific notation for display
pd.set_option('display.float_format', '{:.6f}'.format)
df.describe()

	Count	Zip Code	Latitude	Longitude	Tenure Months	Monthly Charges	Churn Value	Churn Score	CLTV
count	7043.000000	7043.000000	7043.000000	7043.000000	7043.000000	7043.000000	7043.000000	7043.000000	7043.000000
mean	1.000000	93521.964646	36.282441	-119.798880	32.371149	64.761692	0.265370	58.699418	4400.295755
std	0.000000	1865.794555	2.455723	2.157889	24.559481	30.090047	0.441561	21.525131	1183.057152
min	1.000000	90001.000000	32.555828	-124.301372	0.000000	18.250000	0.000000	5.000000	2003.000000
25%	1.000000	92102.000000	34.030915	-121.815412	9.000000	35.500000	0.000000	40.000000	3469.000000
50%	1.000000	93552.000000	36.391777	-119.730885	29.000000	70.350000	0.000000	61.000000	4527.000000
75%	1.000000	95351.000000	38.224869	-118.043237	55.000000	89.850000	1.000000	75.000000	5380.500000
max	1.000000	96161.000000	41.962127	-114.192901	72.000000	118.750000	1.000000	100.000000	6500.000000

pd.reset_option('display.float_format') # Reset format back to default

Preprocessing the data

Preparing the data so that it is suitable for our use case

Remove Uneccesary Columns

Some columns should not be fed into the algorithm as they are useless in predicting customer churn and may reduce the model accuracy. Another reason for this is that for some columns, null may be a valid value (for ex. Churn Reason). Therefore we should remove these to avoid having missing values in our data.

columns_to_drop = ['CustomerID', 'Count', 'Lat Long', 'Zip Code', 'Churn Reason', 'Churn Label']
df = df.drop(columns=columns_to_drop, axis=1)

Handle Missing Values

df = df.dropna() # Drop any rows or columns with null or otherwise missing values
df.head()

	Country	State	City	Latitude	Longitude	Gender	Senior Citizen	Partner	Dependents	Tenure Months	...	Streaming TV	Streaming Movies	Contract	Paperless Billing	Payment Method	Monthly Charges	Total Charges	Churn Value	Churn Score	CLTV
0	United States	California	Los Angeles	33.964131	-118.272783	Male	No	No	No	2	...	No	No	Month-to-month	Yes	Mailed check	53.85	108.15	1	86	3239
1	United States	California	Los Angeles	34.059281	-118.307420	Female	No	No	Yes	2	...	No	No	Month-to-month	Yes	Electronic check	70.70	151.65	1	67	2701
2	United States	California	Los Angeles	34.048013	-118.293953	Female	No	No	Yes	8	...	Yes	Yes	Month-to-month	Yes	Electronic check	99.65	820.5	1	86	5372
3	United States	California	Los Angeles	34.062125	-118.315709	Female	No	Yes	Yes	28	...	Yes	Yes	Month-to-month	Yes	Electronic check	104.80	3046.05	1	84	5003
4	United States	California	Los Angeles	34.039224	-118.266293	Male	No	No	Yes	49	...	Yes	Yes	Month-to-month	Yes	Bank transfer (automatic)	103.70	5036.3	1	89	5340

5 rows × 27 columns

Sanitise Numerical Values

Ensure that numerical values are formatted properly, removing any rows that do not conform to our sanitisation rules

numeric_columns = ['Tenure Months', 'Monthly Charges', 'Total Charges', 'Churn Score', 'Latitude', 'Longitude', 'CLTV']

for col in numeric_columns:
    df[col] = pd.to_numeric(df[col], errors='coerce') # Convert to numeric

df = df.dropna(subset=numeric_columns)

Normalize Numeric Columns

Here we sclae the numeric columns to standardise their range so that large values do not skew the prediction

scaler = StandardScaler()
df[numeric_columns] = scaler.fit_transform(df[numeric_columns])

Convert Categories to Numerical Values

Many of the columns like "Country" and "State" are stored as text values and logistic regression requires numeric input. Here we use one-hot encoding to convert them into binary columns.

df = pd.get_dummies(df, drop_first=True)

df.head()

	Latitude	Longitude	Tenure Months	Monthly Charges	Total Charges	Churn Value	Churn Score	CLTV	City_Acton	City_Adelanto	...	Streaming TV_No internet service	Streaming TV_Yes	Streaming Movies_No internet service	Streaming Movies_Yes	Contract_One year	Contract_Two year	Paperless Billing_Yes	Payment Method_Credit card (automatic)	Payment Method_Electronic check	Payment Method_Mailed check
0	-0.944312	0.707522	-1.239504	-0.363923	-0.959649	1	1.267300	-0.983181	False	False	...	False	False	False	False	False	False	True	False	False	True
1	-0.905569	0.691467	-1.239504	0.196178	-0.940457	1	0.384802	-1.438215	False	False	...	False	False	False	False	False	False	True	False	True	False
2	-0.910157	0.697709	-0.995040	1.158489	-0.645369	1	1.267300	0.820883	False	False	...	False	True	False	True	False	False	True	False	True	False
3	-0.904411	0.687625	-0.180161	1.329677	0.336516	1	1.174405	0.508788	False	False	...	False	True	False	True	False	False	True	False	True	False
4	-0.913735	0.710530	0.675462	1.293113	1.214589	1	1.406642	0.793818	False	False	...	False	True	False	True	False	False	True	False	False	False

5 rows × 1163 columns

df.to_csv("out.csv") # For debugging in excel

Prepare for Training

Define Target and Features

Define the features of the data and the target variable. In this case, our target is Churn Value and our features are all the other columns

X = df.drop(['Churn Value', 'Churn Score', 'CLTV'], axis=1)
y = df['Churn Value']

Split the Data into Train and Test Sets

In order to train the model, we must feed it some data. However, we also need to be able to test the model, so we should save some data for testing. This is to provide fairness in testing as it would be an unfair test to show the model only data that it has already been trained on.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Training the Logistic Regression Model

Here we perform the actual training of the LR model

model = LogisticRegression()
model.fit(X_train, y_train)

LogisticRegression()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Evaluating Model Performance

We have trained the model, so it is now time to test it

y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred))
print("AUC-ROC Score:", roc_auc_score(y_test, y_prob))

              precision    recall  f1-score   support

           0       0.84      0.89      0.87      1012
           1       0.68      0.57      0.62       395

    accuracy                           0.80      1407
   macro avg       0.76      0.73      0.74      1407
weighted avg       0.80      0.80      0.80      1407

AUC-ROC Score: 0.8540451293340671

The above results tell us a few things:

• The target value can be one of two classes; 0 (Non-churn) or 1 (Churn)
• Precision - The precision of cases that are correct, for non churn and churn respectively. In this case, non-churn has better precision
• Recall - The percentage of actual non-churn/churn cases that are identified. In this case, non-churn has better recall.
• F1 - Balance between precision and recall. In this case, non-churn has a better balance of precision/recall
• AUC-ROC Score - The overall performance of the model. A score of 0.5 would be equivalent to a random coin-flip. Therefore the above AUC-ROC score shows that the model performs admirably compared to random chance, however it is not highly accurate.

Interpret Results

My understanding of the results is as follows:

• A higher absolute value of the coefficient means there is a stronger impact on the target variable
• Positive coefficients mean that as the feature increases, the likelihood of churn also increases. The opposite is tru for negative coefficients

feature_importance = pd.DataFrame({'Feature': X.columns, 'Coefficient': model.coef_[0]})
print(feature_importance.sort_values(by='Coefficient', ascending=False))

                Feature  Coefficient
946    City_Smith River     1.528224
874      City_San Dimas     1.407091
1028    City_Travis Afb     1.370443
242      City_Cupertino     1.341841
758    City_Pearblossom     1.329060
...                 ...          ...
60     City_Bakersfield    -1.322815
909    City_Santa Maria    -1.333695
1155  Contract_Two year    -1.385384
2         Tenure Months    -1.478320
1136     Dependents_Yes    -1.512725

[1160 rows x 2 columns]

The first time I trained this model, I noticed that City appeared to have a disproportionate effect on the prediction. After some research, I came to the conclusion that this was an example of overfitting, where too many highly granular categorical values can lead to skewed results. For example, if there is just a single customer based in Smith River who leaves the telco, the model will end up assuming that everyone from Smith River will leave.

This revelation made me re-evaluate how I preprocess the data.

Optimising the Model

In order to try to increase the model's performance, I decided to remove City from the training data, and retrain: (hot-one encoding created lots of City columns, so need to do some additional work to remove those)

# Remove City from the data
columns_to_drop = [col for col in df.columns if col.startswith('City')]
df = df.drop(columns=columns_to_drop)

# Prepare for training, this time without City
X = df.drop(['Churn Value', 'Churn Score', 'CLTV'], axis=1)
y = df['Churn Value']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train the model
model = LogisticRegression()
model.fit(X_train, y_train)

# Test the model
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

# Show the results
print(classification_report(y_test, y_pred))
print("AUC-ROC Score:", roc_auc_score(y_test, y_prob))
feature_importance = pd.DataFrame({'Feature': X.columns, 'Coefficient': model.coef_[0]})
print(feature_importance.sort_values(by='Coefficient', ascending=False))

              precision    recall  f1-score   support

           0       0.84      0.90      0.87      1012
           1       0.70      0.57      0.63       395

    accuracy                           0.81      1407
   macro avg       0.77      0.74      0.75      1407
weighted avg       0.80      0.81      0.81      1407

AUC-ROC Score: 0.862465602641717
                                   Feature  Coefficient
4                            Total Charges     0.730996
28                   Paperless Billing_Yes     0.399350
3                          Monthly Charges     0.384210
12            Internet Service_Fiber optic     0.331254
7                              Partner_Yes     0.306531
30         Payment Method_Electronic check     0.267722
11                      Multiple Lines_Yes     0.159920
6                       Senior Citizen_Yes     0.130189
10         Multiple Lines_No phone service     0.107944
25                    Streaming Movies_Yes     0.055059
5                              Gender_Male    -0.018570
23                        Streaming TV_Yes    -0.023652
0                                 Latitude    -0.030582
1                                Longitude    -0.033503
31             Payment Method_Mailed check    -0.037014
13                     Internet Service_No    -0.066945
20        Tech Support_No internet service    -0.066945
24    Streaming Movies_No internet service    -0.066945
14     Online Security_No internet service    -0.066945
22        Streaming TV_No internet service    -0.066945
16       Online Backup_No internet service    -0.066945
18   Device Protection_No internet service    -0.066945
19                   Device Protection_Yes    -0.080835
29  Payment Method_Credit card (automatic)    -0.182919
17                       Online Backup_Yes    -0.204046
21                        Tech Support_Yes    -0.402609
15                     Online Security_Yes    -0.481805
26                       Contract_One year    -0.636224
9                        Phone Service_Yes    -0.647546
27                       Contract_Two year    -1.337136
8                           Dependents_Yes    -1.483809
2                            Tenure Months    -1.486541

As can be seen in the results, removing City from the data slightly improved the model's AUC-ROC Score. In order to possibly improve the AUC-ROC Score even further, the Cities could be grouped into larger 'regions'.

The above results show that Total Charges have the largest postive correlation with increased customer churn, suggesting that the telco should perhaps offer targeted discounts to those customers who are paying the most. The above results show that Gender has the lowest correlation with churn. The results also show that Tenure Months has the highest negative correlation with customer churn, however this likely comes down to correlation and not causation. The feature with the second highest correlation with churn is Dependents_Yes, suggesting perhaps that customers with dependents have less time or energy to review their outgoings.

Hyperparameter Tuning

Now that the dataset has been thoroughly processed and optimised, we now need to start performing hyperparameter tuning to try to bump up the model's AUC-ROC score even further. First we define our parameter grid, which will be used to generate a list of hyperparameter combinations to tune our model.

# Define the parameter grid
param_grid = {
    'C': [0.01, 0.1, 1, 10, 100],
    'penalty': ['l2'],  # Can also try 'l1' with solvers like 'liblinear'
    'solver': ['liblinear', 'saga']
}

• C (Regularization Strength) - Controls the regularisaion strength. Smaller values mean stronger regularisation. THis can help us correct for overfitting
• Penalty - Specifies the norm used for penalisation
• Solver - Algorithm to use for optimisation

Now we can begin tuning our model:

# Initialize the model
log_reg = LogisticRegression(max_iter=1000)

# GridSearchCV
grid_search = GridSearchCV(estimator=log_reg, param_grid=param_grid, 
                           scoring='roc_auc', cv=5, verbose=1)
grid_search.fit(X_train, y_train)

# Best parameters
print("Best Parameters:", grid_search.best_params_)

# Use the best model
model = grid_search.best_estimator_

Fitting 5 folds for each of 10 candidates, totalling 50 fits
Best Parameters: {'C': 100, 'penalty': 'l2', 'solver': 'liblinear'}

The grid search has discovered what it believes are the best combination of hyperparameters for the given model. Now we will test the model against the previous one.

# Test the model
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

# Show the results
print(classification_report(y_test, y_pred))
print("AUC-ROC Score:", roc_auc_score(y_test, y_prob))

              precision    recall  f1-score   support

           0       0.85      0.91      0.87      1012
           1       0.70      0.58      0.64       395

    accuracy                           0.81      1407
   macro avg       0.78      0.74      0.76      1407
weighted avg       0.81      0.81      0.81      1407

AUC-ROC Score: 0.8626382148396459

Here we can see only a very slight improvement in performance versus the previous iteration of the model. I believe the reason for this is that the default hyperparameter selection already chooses some sensible defaults, meaning the model is mostly optimised as well as it could be for this particular use-case.

Thoughts on LR

Overall, the model performed admirably given the minimal amount of tuning required. However, I believe it is clear that the dataset is not as well optimised as it could be. I would recommend replacing the City column with a broader column that perhaps represents location data at the State or Region level. I recommend this in order to reduce the risk of overfitting. This could be done retroactively by submitting the location data to a geolocation service.

Support Vector Machine

Now that we have evaluated the capability of a Logistic Regression model, we will now look at a Support Vector Machine model. As we have already performed the requisite preprocessing and scaling, we don't need to repeat it.

# Initialize the SVM model with a RBF kernel
model = SVC(kernel='rbf', C=1, gamma='scale', probability=True)

# Train the model
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

# Evaluate the model
print(classification_report(y_test, y_pred))
print("AUC-ROC Score:", roc_auc_score(y_test, y_prob))

              precision    recall  f1-score   support

           0       0.83      0.92      0.88      1012
           1       0.72      0.53      0.61       395

    accuracy                           0.81      1407
   macro avg       0.78      0.73      0.74      1407
weighted avg       0.80      0.81      0.80      1407

AUC-ROC Score: 0.8390603892530144

As can be seen from the results, SVM produces a relatively performant model. The model is not, however, as performant as the LR model we built earlier. I believe the reason for this is that there is not a clearly distinct split in the factors leading to customer churn, meaning SVM struggles to find a clear margin with the default settings. Perhaps hyperperameter tuning will help us achieve better performance?

Tuning the Support Vector Machine Model

Again we will use a grid search approach to hyperparameter tuning:

# Define the parameter grid
param_grid = {
    'C': [0.01, 0.1, 1, 10, 100],         # Regularization strength
    'kernel': ['linear', 'rbf', 'poly'],  # Types of kernels
    'gamma': ['scale', 'auto', 0.01, 0.1],  # For rbf and poly kernels
    'degree': [3, 4, 5]                   # For polynomial kernel, controls degree of polynomial
}

# Initialize the model
svm = SVC(probability=True)  # Enable probability estimates (for ROC-AUC)

# Perform GridSearchCV
grid_search = GridSearchCV(estimator=svm, param_grid=param_grid, scoring='roc_auc', cv=5, verbose=1)
grid_search.fit(X_train, y_train)

# Best parameters
print("Best Parameters:", grid_search.best_params_)

# Use the best model
best_model = grid_search.best_estimator_

# Test the best model
y_pred = best_model.predict(X_test)
y_prob = best_model.predict_proba(X_test)[:, 1]

# Show the results
print(classification_report(y_test, y_pred))
print("AUC-ROC Score:", roc_auc_score(y_test, y_prob))

Ah... That's not quite right. SVM training takes significantly longer than LR training, and there are more hyperparameters to tune. The result is that the above code takes a significant amount of time to complete. (Slow code commented out)

To work around this, we will swap out the Grid Search for a Randomised Search. This results in a significantly smaller pool of combinations to try, at the expense of potentially leaving some model performance on the table.

# Define the parameter distribution for random search
param_dist = {
    'C': np.logspace(-2, 2, 5),      # Regularization strength from 0.01 to 100
    'kernel': ['linear', 'rbf', 'poly'],  # Types of kernels
    'gamma': ['scale', 'auto'] + np.logspace(-2, 1, 5).tolist(),  # Gamma values
    'degree': [3, 4, 5]  # For polynomial kernel
}

# Initialize the SVM model
svm = SVC(probability=True)

# RandomizedSearchCV
random_search = RandomizedSearchCV(estimator=svm, param_distributions=param_dist, 
                                   n_iter=50, cv=5, scoring='roc_auc', verbose=1, random_state=42)
random_search.fit(X_train, y_train)  # Use the preprocessed (scaled) data

# Best parameters found by RandomizedSearchCV
print("Best Parameters:", random_search.best_params_)

# Use the best model
best_model = random_search.best_estimator_

# Test the model
y_pred = best_model.predict(X_test)  # Use the preprocessed (scaled) data
y_prob = best_model.predict_proba(X_test)[:, 1]  # Use the preprocessed (scaled) data

# Show the results
print(classification_report(y_test, y_pred))
print("AUC-ROC Score:", roc_auc_score(y_test, y_prob))

Hmm. Still incredibly slow. I'll adjust the ratio of training to testing data to see if that makes it quicker.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.8, random_state=42)

# RandomizedSearchCV
random_search = RandomizedSearchCV(estimator=svm, param_distributions=param_dist, 
                                   n_iter=50, cv=5, scoring='roc_auc', verbose=1, random_state=42)
random_search.fit(X_train, y_train)  # Use the preprocessed (scaled) data

# Best parameters found by RandomizedSearchCV
print("Best Parameters:", random_search.best_params_)

# Use the best model
best_model = random_search.best_estimator_

# Test the model
y_pred = best_model.predict(X_test)  # Use the preprocessed (scaled) data
y_prob = best_model.predict_proba(X_test)[:, 1]  # Use the preprocessed (scaled) data

# Show the results
print(classification_report(y_test, y_pred))
print("AUC-ROC Score:", roc_auc_score(y_test, y_prob))

Fitting 5 folds for each of 50 candidates, totalling 250 fits
Best Parameters: {'kernel': 'linear', 'gamma': 1.7782794100389228, 'degree': 3, 'C': 0.1}
              precision    recall  f1-score   support

           0       0.84      0.90      0.87      4100
           1       0.67      0.53      0.59      1526

    accuracy                           0.80      5626
   macro avg       0.75      0.72      0.73      5626
weighted avg       0.79      0.80      0.79      5626

AUC-ROC Score: 0.8417052392673338

With less training data, the hyperparameter tuning completes much faster. Now lets increase the amount of training data used once more, with our tuned hyperparameters in play:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize the SVM model with the identified best hyperparameters
model = SVC(kernel='linear', C=0.1, gamma=1.7782794100389228, probability=True)

# Train the model
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

# Evaluate the model
print(classification_report(y_test, y_pred))
print("AUC-ROC Score:", roc_auc_score(y_test, y_prob))

              precision    recall  f1-score   support

           0       0.84      0.90      0.87      1012
           1       0.69      0.57      0.63       395

    accuracy                           0.81      1407
   macro avg       0.77      0.74      0.75      1407
weighted avg       0.80      0.81      0.80      1407

AUC-ROC Score: 0.8553960074048131

Success! Quite a sizeable improvement in model performance compared to our starting SVM. However, it is still less performant than our optimised and tuned LR model. In addition, hyperparameter tuning took significantly longer for SVM, even using a Random Search and a reduced training set size. Therefore, I would recommend using an LR model for this particular type of problem, with this size of dataset.

TODO: Limitations