In [23]:
# Creating histograms
df[num_cols].hist(figsize=(14,14))
plt.show()
XYZ is an MNC that has thousands of employees spread across the globe. The company believes in hiring the best talent available and retaining them for as long as possible. A huge amount of resources is spent on retaining existing employees through various initiatives. The Head of People Operations wants to bring down the cost of retaining employees. For this, he proposes limiting the incentives to only those employees who are at risk of attrition. As a recently hired Data Scientist in the People Operations Department, you have been asked to identify patterns in characteristics of employees who leave the organization. Also, you have to use this information to predict if an employee is at risk of attrition. This information will be used to target them with incentives.
The data contains information on employees' demographic details, work-related metrics, and attrition flag.
In the real world, you will not find definitions for some of your variables. It is the part of the analysis to figure out what they might mean.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# To scale the data using z-score
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
# Algorithms to use
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
# Metrics to evaluate the model
from sklearn import metrics
from sklearn.metrics import confusion_matrix, classification_report, precision_recall_curve,recall_score
# To ignore warnings
import warnings
warnings.filterwarnings("ignore")
# Loading the dataset
df = pd.read_excel('HR_Employee_Attrition_Dataset.xlsx')
df.head()
| EmployeeNumber | Attrition | Age | BusinessTravel | DailyRate | Department | DistanceFromHome | Education | EducationField | EnvironmentSatisfaction | ... | RelationshipSatisfaction | StandardHours | StockOptionLevel | TotalWorkingYears | TrainingTimesLastYear | WorkLifeBalance | YearsAtCompany | YearsInCurrentRole | YearsSinceLastPromotion | YearsWithCurrManager | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | Yes | 41 | Travel_Rarely | 1102 | Sales | 1 | 2 | Life Sciences | 2 | ... | 1 | 80 | 0 | 8 | 0 | 1 | 6 | 4 | 0 | 5 |
| 1 | 2 | No | 49 | Travel_Frequently | 279 | Research & Development | 8 | 1 | Life Sciences | 3 | ... | 4 | 80 | 1 | 10 | 3 | 3 | 10 | 7 | 1 | 7 |
| 2 | 3 | Yes | 37 | Travel_Rarely | 1373 | Research & Development | 2 | 2 | Other | 4 | ... | 2 | 80 | 0 | 7 | 3 | 3 | 0 | 0 | 0 | 0 |
| 3 | 4 | No | 33 | Travel_Frequently | 1392 | Research & Development | 3 | 4 | Life Sciences | 4 | ... | 3 | 80 | 0 | 8 | 3 | 3 | 8 | 7 | 3 | 0 |
| 4 | 5 | No | 27 | Travel_Rarely | 591 | Research & Development | 2 | 1 | Medical | 1 | ... | 4 | 80 | 1 | 6 | 3 | 3 | 2 | 2 | 2 | 2 |
5 rows × 34 columns
# Let us see the info of the data
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 2940 entries, 0 to 2939 Data columns (total 34 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 EmployeeNumber 2940 non-null int64 1 Attrition 2940 non-null object 2 Age 2940 non-null int64 3 BusinessTravel 2940 non-null object 4 DailyRate 2940 non-null int64 5 Department 2940 non-null object 6 DistanceFromHome 2940 non-null int64 7 Education 2940 non-null int64 8 EducationField 2940 non-null object 9 EnvironmentSatisfaction 2940 non-null int64 10 Gender 2940 non-null object 11 HourlyRate 2940 non-null int64 12 JobInvolvement 2940 non-null int64 13 JobLevel 2940 non-null int64 14 JobRole 2940 non-null object 15 JobSatisfaction 2940 non-null int64 16 MaritalStatus 2940 non-null object 17 MonthlyIncome 2940 non-null int64 18 MonthlyRate 2940 non-null int64 19 NumCompaniesWorked 2940 non-null int64 20 Over18 2940 non-null object 21 OverTime 2940 non-null object 22 PercentSalaryHike 2940 non-null int64 23 PerformanceRating 2940 non-null int64 24 RelationshipSatisfaction 2940 non-null int64 25 StandardHours 2940 non-null int64 26 StockOptionLevel 2940 non-null int64 27 TotalWorkingYears 2940 non-null int64 28 TrainingTimesLastYear 2940 non-null int64 29 WorkLifeBalance 2940 non-null int64 30 YearsAtCompany 2940 non-null int64 31 YearsInCurrentRole 2940 non-null int64 32 YearsSinceLastPromotion 2940 non-null int64 33 YearsWithCurrManager 2940 non-null int64 dtypes: int64(25), object(9) memory usage: 781.1+ KB
Observations:
Let's check the unique values in each column
# Checking unique values in each column
df.nunique()
EmployeeNumber 2940 Attrition 2 Age 43 BusinessTravel 3 DailyRate 886 Department 3 DistanceFromHome 29 Education 5 EducationField 6 EnvironmentSatisfaction 4 Gender 2 HourlyRate 71 JobInvolvement 4 JobLevel 5 JobRole 9 JobSatisfaction 4 MaritalStatus 3 MonthlyIncome 1349 MonthlyRate 1427 NumCompaniesWorked 10 Over18 1 OverTime 2 PercentSalaryHike 15 PerformanceRating 2 RelationshipSatisfaction 4 StandardHours 1 StockOptionLevel 4 TotalWorkingYears 40 TrainingTimesLastYear 7 WorkLifeBalance 4 YearsAtCompany 37 YearsInCurrentRole 19 YearsSinceLastPromotion 16 YearsWithCurrManager 18 dtype: int64
Observations:
Let's drop the columns mentioned above and define lists for numerical and categorical columns to explore them separately.
# Dropping the columns
df=df.drop(['EmployeeNumber','Over18','StandardHours'],axis=1) # axis 1 means column
# Creating numerical columns
num_cols=['DailyRate','Age','DistanceFromHome','MonthlyIncome','MonthlyRate','PercentSalaryHike','TotalWorkingYears',
'YearsAtCompany','NumCompaniesWorked','HourlyRate',
'YearsInCurrentRole','YearsSinceLastPromotion','YearsWithCurrManager','TrainingTimesLastYear']
# Creating categorical variables
cat_cols= ['Attrition','OverTime','BusinessTravel', 'Department','Education', 'EducationField','JobSatisfaction','EnvironmentSatisfaction','WorkLifeBalance',
'StockOptionLevel','Gender', 'PerformanceRating', 'JobInvolvement','JobLevel', 'JobRole', 'MaritalStatus','RelationshipSatisfaction']
# Checking summary statistics
df[num_cols].describe().T
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| DailyRate | 2940.0 | 802.485714 | 403.440447 | 102.0 | 465.0 | 802.0 | 1157.0 | 1499.0 |
| Age | 2940.0 | 36.923810 | 9.133819 | 18.0 | 30.0 | 36.0 | 43.0 | 60.0 |
| DistanceFromHome | 2940.0 | 9.192517 | 8.105485 | 1.0 | 2.0 | 7.0 | 14.0 | 29.0 |
| MonthlyIncome | 2940.0 | 6502.931293 | 4707.155770 | 1009.0 | 2911.0 | 4919.0 | 8380.0 | 19999.0 |
| MonthlyRate | 2940.0 | 14313.103401 | 7116.575021 | 2094.0 | 8045.0 | 14235.5 | 20462.0 | 26999.0 |
| PercentSalaryHike | 2940.0 | 15.209524 | 3.659315 | 11.0 | 12.0 | 14.0 | 18.0 | 25.0 |
| TotalWorkingYears | 2940.0 | 11.279592 | 7.779458 | 0.0 | 6.0 | 10.0 | 15.0 | 40.0 |
| YearsAtCompany | 2940.0 | 7.008163 | 6.125483 | 0.0 | 3.0 | 5.0 | 9.0 | 40.0 |
| NumCompaniesWorked | 2940.0 | 2.693197 | 2.497584 | 0.0 | 1.0 | 2.0 | 4.0 | 9.0 |
| HourlyRate | 2940.0 | 65.891156 | 20.325969 | 30.0 | 48.0 | 66.0 | 84.0 | 100.0 |
| YearsInCurrentRole | 2940.0 | 4.229252 | 3.622521 | 0.0 | 2.0 | 3.0 | 7.0 | 18.0 |
| YearsSinceLastPromotion | 2940.0 | 2.187755 | 3.221882 | 0.0 | 0.0 | 1.0 | 3.0 | 15.0 |
| YearsWithCurrManager | 2940.0 | 4.123129 | 3.567529 | 0.0 | 2.0 | 3.0 | 7.0 | 17.0 |
| TrainingTimesLastYear | 2940.0 | 2.799320 | 1.289051 | 0.0 | 2.0 | 3.0 | 3.0 | 6.0 |
Observations:
Let's explore these variables in some more depth by observing their distributions
# Creating histograms
df[num_cols].hist(figsize=(14,14))
plt.show()
Observations:
The age distribution is close to a normal distribution with the majority of employees between the ages of 25 and 50.
The percentage salary hike is skewed to the right, implying that employees are obtaining smaller increases.
MonthlyIncome and TotalWorkingYears are skewed to the right, indicating that the majority of workers are in entry / mid-level positions in the organization.
DistanceFromHome also has a right skewed distribution, meaning most employees live close to work but there are a few that live further away.
On average, an employee has worked at 2.5 companies. Most employees have worked at only 1 company.
The YearsAtCompany variable distribution shows a good proportion of workers with 10+ years, indicating a significant number of loyal employees at the organization.
The YearsInCurrentRole distribution has three peaks at 0, 2, and 7. There are a few employees that have even stayed in the same role for 15 years and more.
The YearsSinceLastPromotion variable distribution indicates that some employees have not received a promotion in 10-15 years and are still working in the organization. These employees are assumed to be high work-experience employees in upper-management roles, such as co-founders, C-suite employees etc.
The distributions of DailyRate, HourlyRate and MonthlyRate appear to be uniform and do not provide much information. It could be that daily rate refers to the income earned per extra day worked while hourly rate could refer to the same concept applied for extra hours worked per day. Since these rates tend to be broadly similar for multiple employees in the same department, that explains the uniform distribution they show.
# Printing the % sub categories of each category
for i in cat_cols:
print(df[i].value_counts (normalize=True))
print('*'*40)
Attrition No 0.838776 Yes 0.161224 Name: proportion, dtype: float64 **************************************** OverTime No 0.717007 Yes 0.282993 Name: proportion, dtype: float64 **************************************** BusinessTravel Travel_Rarely 0.709524 Travel_Frequently 0.188435 Non-Travel 0.102041 Name: proportion, dtype: float64 **************************************** Department Research & Development 0.653741 Sales 0.303401 Human Resources 0.042857 Name: proportion, dtype: float64 **************************************** Education 3 0.389116 4 0.270748 2 0.191837 1 0.115646 5 0.032653 Name: proportion, dtype: float64 **************************************** EducationField Life Sciences 0.412245 Medical 0.315646 Marketing 0.108163 Technical Degree 0.089796 Other 0.055782 Human Resources 0.018367 Name: proportion, dtype: float64 **************************************** JobSatisfaction 4 0.312245 3 0.300680 1 0.196599 2 0.190476 Name: proportion, dtype: float64 **************************************** EnvironmentSatisfaction 3 0.308163 4 0.303401 2 0.195238 1 0.193197 Name: proportion, dtype: float64 **************************************** WorkLifeBalance 3 0.607483 2 0.234014 4 0.104082 1 0.054422 Name: proportion, dtype: float64 **************************************** StockOptionLevel 0 0.429252 1 0.405442 2 0.107483 3 0.057823 Name: proportion, dtype: float64 **************************************** Gender Male 0.6 Female 0.4 Name: proportion, dtype: float64 **************************************** PerformanceRating 3 0.846259 4 0.153741 Name: proportion, dtype: float64 **************************************** JobInvolvement 3 0.590476 2 0.255102 4 0.097959 1 0.056463 Name: proportion, dtype: float64 **************************************** JobLevel 1 0.369388 2 0.363265 3 0.148299 4 0.072109 5 0.046939 Name: proportion, dtype: float64 **************************************** JobRole Sales Executive 0.221769 Research Scientist 0.198639 Laboratory Technician 0.176190 Manufacturing Director 0.098639 Healthcare Representative 0.089116 Manager 0.069388 Sales Representative 0.056463 Research Director 0.054422 Human Resources 0.035374 Name: proportion, dtype: float64 **************************************** MaritalStatus Married 0.457823 Single 0.319728 Divorced 0.222449 Name: proportion, dtype: float64 **************************************** RelationshipSatisfaction 3 0.312245 4 0.293878 2 0.206122 1 0.187755 Name: proportion, dtype: float64 ****************************************
Observations:
We have analyzed different categorical and numerical variables. Let's now check how does attrition rate is related with other categorical variables.
for i in cat_cols:
if i!='Attrition':
(pd.crosstab(df[i],df['Attrition'],normalize='index')*100).plot(kind='bar',figsize=(8,4),stacked=True)
plt.ylabel('Percentage Attrition %')
Observations:
Let's check the relationship between attrition and Numerical variables
# Mean of numerical variables grouped by attrition
df.groupby(['Attrition'])[num_cols].mean()
| DailyRate | Age | DistanceFromHome | MonthlyIncome | MonthlyRate | PercentSalaryHike | TotalWorkingYears | YearsAtCompany | NumCompaniesWorked | HourlyRate | YearsInCurrentRole | YearsSinceLastPromotion | YearsWithCurrManager | TrainingTimesLastYear | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Attrition | ||||||||||||||
| No | 812.504461 | 37.561233 | 8.915653 | 6832.739659 | 14265.779400 | 15.231144 | 11.862936 | 7.369019 | 2.645580 | 65.952149 | 4.484185 | 2.234388 | 4.367397 | 2.832928 |
| Yes | 750.362869 | 33.607595 | 10.632911 | 4787.092827 | 14559.308017 | 15.097046 | 8.244726 | 5.130802 | 2.940928 | 65.573840 | 2.902954 | 1.945148 | 2.852321 | 2.624473 |
Observations:
We have found out what kind of employees are leaving the company.
# Plotting the correlation between numerical variables
plt.figure(figsize=(15,8))
sns.heatmap(df[num_cols].corr(), annot=True, fmt='0.2f', cmap='YlOrRd')
<Axes: >
Observations:
Now we have explored our data. Let's build the model
Creating dummy variables for categorical Variables
# Creating list of dummy columns
to_get_dummies_for = ['BusinessTravel', 'Department','Education', 'EducationField','EnvironmentSatisfaction', 'Gender', 'JobInvolvement','JobLevel', 'JobRole', 'MaritalStatus' ]
# Creating dummy variables
df = pd.get_dummies(data = df, columns = to_get_dummies_for, drop_first = True)
# Mapping overtime and attrition
dict_OverTime = {'Yes': 1, 'No':0}
dict_attrition = {'Yes': 1, 'No': 0}
df['OverTime'] = df.OverTime.map(dict_OverTime)
df['Attrition'] = df.Attrition.map(dict_attrition)
Separating the independent variables (X) and the dependent variable (Y)
# Separating target variable and other variables
Y= df.Attrition
X= df.drop(columns = ['Attrition'])
Splitting the data into 70% train and 30% test set
Some classification problems can exhibit a large imbalance in the distribution of the target classes: for instance there could be several times more negative samples than positive samples. In such cases it is recommended to use the stratified sampling technique to ensure that relative class frequencies are approximately preserved in each train and validation fold.
# Splitting the data
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.3, random_state = 1, stratify = Y)
The independent variables in this dataset have different scales. When features have different scales from each other, there is a chance that a higher weightage will be given to features that have a higher magnitude, and they will dominate over other features whose magnitude changes may be smaller but whose percentage changes may be just as significant or even larger. This will impact the performance of our machine learning algorithm, and we do not want our algorithm to be biased towards one feature.
The solution to this issue is Feature Scaling, i.e. scaling the dataset so as to give every transformed variable a comparable scale.
Tree based models such as Decision Trees and Random Forest does not require feature scaling to be performed as they are not sensitive to the variance in the data.
We will scale the data for Logistic Regression and SVM. We will use the Standard Scaler method, which centers and scales the dataset using the Z-Score. It standardizes features by subtracting the mean and scaling it to have unit variance. The standard score of sample x is calculated as:
z = (x - u) / s
where u is the mean of the training samples (zero) and s is the standard deviation of the training samples.
# Scaling the data
sc=StandardScaler()
# Fit_transform on train data
X_train_scaled=sc.fit_transform(X_train)
X_train_scaled=pd.DataFrame(X_train_scaled, columns=X.columns)
# Transform on test data
X_test_scaled=sc.transform(X_test)
X_test_scaled=pd.DataFrame(X_test_scaled, columns=X.columns)
The model can make two types of wrong predictions:
Which case is more important?
How to reduce this loss i.e the need to reduce False Negatives?
Also, let's create a function to calculate and print the classification report and confusion matrix so that we don't have to rewrite the same code repeatedly for each model.
# Creating metric function
def metrics_score(actual, predicted):
print(classification_report(actual, predicted))
cm = confusion_matrix(actual, predicted)
plt.figure(figsize=(8,5))
sns.heatmap(cm, annot=True, fmt='.2f', xticklabels=['Not Attrite', 'Attrite'], yticklabels=['Not Attrite', 'Attrite'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
We will be building 4 different models:
Logistic Regression is a supervised learning algorithm which is used for binary classification problems i.e. where the dependent variable is categorical and has only two possible values. In logistic regression, we use the sigmoid function to calculate the probability of an event y, given some features x as:
P(y)=1/exp(1 + exp(-x))# Fitting logistic regression model
lg=LogisticRegression()
lg.fit(X_train_scaled,y_train)
LogisticRegression()In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
LogisticRegression()
Let's check the model performance
# Checking the performance on the training data
y_pred_train = lg.predict(X_train_scaled)
metrics_score(y_train, y_pred_train)
precision recall f1-score support
0 0.91 0.98 0.94 1726
1 0.81 0.50 0.62 332
accuracy 0.90 2058
macro avg 0.86 0.74 0.78 2058
weighted avg 0.89 0.90 0.89 2058
Reading the confusion matrix (clockwise):
True Negative (Actual=0, Predicted=0): Model predicts that an employee would not attrite and the employee does not attrite
False Positive (Actual=0, Predicted=1): Model predicts that an employee would attrite but the employee does not attrite
False Negative (Actual=1, Predicted=0): Model predicts that an employee would not attrite but the employee attrites
True Positive (Actual=1, Predicted=1): Model predicts that an employee would attrite and the employee actually attrites
# Checking the performance on the test dataset
y_pred_test = lg.predict(X_test_scaled)
metrics_score(y_test, y_pred_test)
precision recall f1-score support
0 0.90 0.97 0.94 740
1 0.76 0.46 0.58 142
accuracy 0.89 882
macro avg 0.83 0.72 0.76 882
weighted avg 0.88 0.89 0.88 882
Observations:
Let's check the coefficients and find which variables are leading to attrition and which can help to reduce the attrition
# Printing the coefficients of logistic regression
cols=X.columns
coef_lg=lg.coef_
pd.DataFrame(coef_lg,columns=cols).T.sort_values(by = 0,ascending = False)
| 0 | |
|---|---|
| OverTime | 0.959309 |
| BusinessTravel_Travel_Frequently | 0.715550 |
| MaritalStatus_Single | 0.614826 |
| YearsSinceLastPromotion | 0.547480 |
| YearsAtCompany | 0.519314 |
| NumCompaniesWorked | 0.499629 |
| Department_Sales | 0.483409 |
| Department_Research & Development | 0.481904 |
| BusinessTravel_Travel_Rarely | 0.440191 |
| DistanceFromHome | 0.385265 |
| JobRole_Sales Executive | 0.383569 |
| MaritalStatus_Married | 0.288594 |
| JobRole_Human Resources | 0.277871 |
| JobLevel_5 | 0.271247 |
| JobRole_Laboratory Technician | 0.177542 |
| JobRole_Sales Representative | 0.174790 |
| Gender_Male | 0.165442 |
| Education_3 | 0.158543 |
| Education_2 | 0.130707 |
| Education_4 | 0.113111 |
| JobRole_Manufacturing Director | 0.110652 |
| Education_5 | 0.092586 |
| EducationField_Technical Degree | 0.082834 |
| MonthlyRate | 0.059602 |
| HourlyRate | 0.047864 |
| JobLevel_3 | 0.006875 |
| EducationField_Marketing | -0.013985 |
| JobRole_Manager | -0.032688 |
| PerformanceRating | -0.032814 |
| PercentSalaryHike | -0.075219 |
| DailyRate | -0.095761 |
| StockOptionLevel | -0.109710 |
| EducationField_Other | -0.134466 |
| JobLevel_4 | -0.160350 |
| WorkLifeBalance | -0.211983 |
| JobRole_Research Scientist | -0.234910 |
| TrainingTimesLastYear | -0.242383 |
| Age | -0.274531 |
| RelationshipSatisfaction | -0.312382 |
| EducationField_Life Sciences | -0.319328 |
| JobRole_Research Director | -0.353661 |
| EducationField_Medical | -0.354183 |
| JobSatisfaction | -0.375085 |
| YearsWithCurrManager | -0.381770 |
| YearsInCurrentRole | -0.431586 |
| EnvironmentSatisfaction_2 | -0.451521 |
| JobInvolvement_2 | -0.481784 |
| EnvironmentSatisfaction_3 | -0.497815 |
| TotalWorkingYears | -0.499453 |
| MonthlyIncome | -0.599360 |
| EnvironmentSatisfaction_4 | -0.651756 |
| JobInvolvement_4 | -0.654034 |
| JobLevel_2 | -0.713244 |
| JobInvolvement_3 | -0.749442 |
Observations:
Features which positively affect on the attrition rate are:
Features which negatively affect on the attrition rate are:
Observations:
The coefficients of the logistic regression model give us the log of odds, which is hard to interpret in the real world. We can convert the log of odds into real odds by taking its exponential.
# Finding the odds
odds = np.exp(lg.coef_[0])
# Adding the odds to a dataframe and sorting the values
pd.DataFrame(odds, X_train_scaled.columns, columns = ['odds']).sort_values(by ='odds', ascending = False)
| odds | |
|---|---|
| OverTime | 2.609893 |
| BusinessTravel_Travel_Frequently | 2.045312 |
| MaritalStatus_Single | 1.849334 |
| YearsSinceLastPromotion | 1.728891 |
| YearsAtCompany | 1.680873 |
| NumCompaniesWorked | 1.648110 |
| Department_Sales | 1.621594 |
| Department_Research & Development | 1.619154 |
| BusinessTravel_Travel_Rarely | 1.553004 |
| DistanceFromHome | 1.470003 |
| JobRole_Sales Executive | 1.467513 |
| MaritalStatus_Married | 1.334550 |
| JobRole_Human Resources | 1.320316 |
| JobLevel_5 | 1.311599 |
| JobRole_Laboratory Technician | 1.194278 |
| JobRole_Sales Representative | 1.190996 |
| Gender_Male | 1.179914 |
| Education_3 | 1.171802 |
| Education_2 | 1.139634 |
| Education_4 | 1.119757 |
| JobRole_Manufacturing Director | 1.117006 |
| Education_5 | 1.097008 |
| EducationField_Technical Degree | 1.086361 |
| MonthlyRate | 1.061414 |
| HourlyRate | 1.049028 |
| JobLevel_3 | 1.006899 |
| EducationField_Marketing | 0.986113 |
| JobRole_Manager | 0.967840 |
| PerformanceRating | 0.967718 |
| PercentSalaryHike | 0.927540 |
| DailyRate | 0.908681 |
| StockOptionLevel | 0.896094 |
| EducationField_Other | 0.874182 |
| JobLevel_4 | 0.851846 |
| WorkLifeBalance | 0.808978 |
| JobRole_Research Scientist | 0.790642 |
| TrainingTimesLastYear | 0.784756 |
| Age | 0.759928 |
| RelationshipSatisfaction | 0.731702 |
| EducationField_Life Sciences | 0.726637 |
| JobRole_Research Director | 0.702113 |
| EducationField_Medical | 0.701746 |
| JobSatisfaction | 0.687231 |
| YearsWithCurrManager | 0.682652 |
| YearsInCurrentRole | 0.649479 |
| EnvironmentSatisfaction_2 | 0.636659 |
| JobInvolvement_2 | 0.617680 |
| EnvironmentSatisfaction_3 | 0.607857 |
| TotalWorkingYears | 0.606863 |
| MonthlyIncome | 0.549163 |
| EnvironmentSatisfaction_4 | 0.521130 |
| JobInvolvement_4 | 0.519944 |
| JobLevel_2 | 0.490052 |
| JobInvolvement_3 | 0.472630 |
Observations
Precision-Recall curves summarize the trade-off between the true positive rate and the positive predictive value for a predictive model using different probability thresholds.
# Predict_proba gives the probability of each observation belonging to each class
y_scores_lg=lg.predict_proba(X_train_scaled)
precisions_lg, recalls_lg, thresholds_lg = precision_recall_curve(y_train, y_scores_lg[:,1])
# Plot values of precisions, recalls, and thresholds
plt.figure(figsize=(10,7))
plt.plot(thresholds_lg, precisions_lg[:-1], 'b--', label='precision')
plt.plot(thresholds_lg, recalls_lg[:-1], 'g--', label = 'recall')
plt.xlabel('Threshold')
plt.legend(loc='upper left')
plt.ylim([0,1])
plt.show()
Observation:
Let's find out the performance of the model at this threshold
optimal_threshold=.35
y_pred_train = lg.predict_proba(X_train_scaled)
metrics_score(y_train, y_pred_train[:,1]>optimal_threshold)
precision recall f1-score support
0 0.93 0.93 0.93 1726
1 0.65 0.64 0.64 332
accuracy 0.89 2058
macro avg 0.79 0.79 0.79 2058
weighted avg 0.89 0.89 0.89 2058
Observations
optimal_threshold=.35
y_pred_test = lg.predict_proba(X_test_scaled)
metrics_score(y_test, y_pred_test[:,1]>optimal_threshold)
precision recall f1-score support
0 0.93 0.93 0.93 740
1 0.62 0.63 0.63 142
accuracy 0.88 882
macro avg 0.78 0.78 0.78 882
weighted avg 0.88 0.88 0.88 882
Observation:
Let's build the models using the two of the widely used kernel functions:
# Fitting SVM
svm = SVC(kernel='linear') # Linear kernal or linear decision boundary
model = svm.fit(X= X_train_scaled, y = y_train)
y_pred_train_svm = model.predict(X_train_scaled)
metrics_score(y_train, y_pred_train_svm)
precision recall f1-score support
0 0.91 0.98 0.94 1726
1 0.81 0.50 0.62 332
accuracy 0.90 2058
macro avg 0.86 0.74 0.78 2058
weighted avg 0.89 0.90 0.89 2058
# Checking performance on the test data
y_pred_test_svm = model.predict(X_test_scaled)
metrics_score(y_test, y_pred_test_svm)
precision recall f1-score support
0 0.90 0.97 0.94 740
1 0.76 0.44 0.56 142
accuracy 0.89 882
macro avg 0.83 0.71 0.75 882
weighted avg 0.88 0.89 0.88 882
svm_rbf=SVC(kernel='rbf',probability=True)
# Fit the model
svm_rbf.fit(X_train_scaled,y_train)
# Predict on train data
y_scores_svm=svm_rbf.predict_proba(X_train_scaled)
precisions_svm, recalls_svm, thresholds_svm = precision_recall_curve(y_train, y_scores_svm[:,1])
# Plot values of precisions, recalls, and thresholds
plt.figure(figsize=(10,7))
plt.plot(thresholds_svm, precisions_svm[:-1], 'b--', label='precision')
plt.plot(thresholds_svm, recalls_svm[:-1], 'g--', label = 'recall')
plt.xlabel('Threshold')
plt.legend(loc='upper left')
plt.ylim([0,1])
plt.show()
optimal_threshold_svm=.35
y_pred_train = svm_rbf.predict_proba(X_train_scaled)
metrics_score(y_train, y_pred_train[:,1]>optimal_threshold_svm)
precision recall f1-score support
0 0.98 0.99 0.98 1726
1 0.92 0.90 0.91 332
accuracy 0.97 2058
macro avg 0.95 0.94 0.95 2058
weighted avg 0.97 0.97 0.97 2058
optimal_threshold_svm=.35
y_pred_test = svm_rbf.predict_proba(X_test_scaled)
metrics_score(y_test, y_pred_test[:,1]>optimal_threshold_svm)
precision recall f1-score support
0 0.97 0.94 0.95 740
1 0.74 0.82 0.78 142
accuracy 0.93 882
macro avg 0.85 0.88 0.87 882
weighted avg 0.93 0.93 0.93 882
Observations:
Let's look at the feature importance of this model and try to analyze why this is happening.