Installing the libraries¶
In [ ]:
pip install pandas numpy scikit-learn
In [19]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
Loading the Dataset¶
In [20]:
df=pd.read_csv('kdd_train.csv')
df.head()
Out[20]:
| duration | protocol_type | service | flag | src_bytes | dst_bytes | land | wrong_fragment | urgent | hot | ... | dst_host_srv_count | dst_host_same_srv_rate | dst_host_diff_srv_rate | dst_host_same_src_port_rate | dst_host_srv_diff_host_rate | dst_host_serror_rate | dst_host_srv_serror_rate | dst_host_rerror_rate | dst_host_srv_rerror_rate | labels | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | tcp | ftp_data | SF | 491 | 0 | 0 | 0 | 0 | 0 | ... | 25 | 0.17 | 0.03 | 0.17 | 0.00 | 0.00 | 0.00 | 0.05 | 0.00 | normal |
| 1 | 0 | udp | other | SF | 146 | 0 | 0 | 0 | 0 | 0 | ... | 1 | 0.00 | 0.60 | 0.88 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | normal |
| 2 | 0 | tcp | private | S0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 26 | 0.10 | 0.05 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | neptune |
| 3 | 0 | tcp | http | SF | 232 | 8153 | 0 | 0 | 0 | 0 | ... | 255 | 1.00 | 0.00 | 0.03 | 0.04 | 0.03 | 0.01 | 0.00 | 0.01 | normal |
| 4 | 0 | tcp | http | SF | 199 | 420 | 0 | 0 | 0 | 0 | ... | 255 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | normal |
5 rows × 42 columns
Checking the info of the dataset¶
In [21]:
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 125973 entries, 0 to 125972 Data columns (total 42 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 duration 125973 non-null int64 1 protocol_type 125973 non-null object 2 service 125973 non-null object 3 flag 125973 non-null object 4 src_bytes 125973 non-null int64 5 dst_bytes 125973 non-null int64 6 land 125973 non-null int64 7 wrong_fragment 125973 non-null int64 8 urgent 125973 non-null int64 9 hot 125973 non-null int64 10 num_failed_logins 125973 non-null int64 11 logged_in 125973 non-null int64 12 num_compromised 125973 non-null int64 13 root_shell 125973 non-null int64 14 su_attempted 125973 non-null int64 15 num_root 125973 non-null int64 16 num_file_creations 125973 non-null int64 17 num_shells 125973 non-null int64 18 num_access_files 125973 non-null int64 19 num_outbound_cmds 125973 non-null int64 20 is_host_login 125973 non-null int64 21 is_guest_login 125973 non-null int64 22 count 125973 non-null int64 23 srv_count 125973 non-null int64 24 serror_rate 125973 non-null float64 25 srv_serror_rate 125973 non-null float64 26 rerror_rate 125973 non-null float64 27 srv_rerror_rate 125973 non-null float64 28 same_srv_rate 125973 non-null float64 29 diff_srv_rate 125973 non-null float64 30 srv_diff_host_rate 125973 non-null float64 31 dst_host_count 125973 non-null int64 32 dst_host_srv_count 125973 non-null int64 33 dst_host_same_srv_rate 125973 non-null float64 34 dst_host_diff_srv_rate 125973 non-null float64 35 dst_host_same_src_port_rate 125973 non-null float64 36 dst_host_srv_diff_host_rate 125973 non-null float64 37 dst_host_serror_rate 125973 non-null float64 38 dst_host_srv_serror_rate 125973 non-null float64 39 dst_host_rerror_rate 125973 non-null float64 40 dst_host_srv_rerror_rate 125973 non-null float64 41 labels 125973 non-null object dtypes: float64(15), int64(23), object(4) memory usage: 40.4+ MB
Finding how many missing values each column has¶
In [23]:
df.isnull().sum()
Out[23]:
duration 0 protocol_type 0 service 0 flag 0 src_bytes 0 dst_bytes 0 land 0 wrong_fragment 0 urgent 0 hot 0 num_failed_logins 0 logged_in 0 num_compromised 0 root_shell 0 su_attempted 0 num_root 0 num_file_creations 0 num_shells 0 num_access_files 0 num_outbound_cmds 0 is_host_login 0 is_guest_login 0 count 0 srv_count 0 serror_rate 0 srv_serror_rate 0 rerror_rate 0 srv_rerror_rate 0 same_srv_rate 0 diff_srv_rate 0 srv_diff_host_rate 0 dst_host_count 0 dst_host_srv_count 0 dst_host_same_srv_rate 0 dst_host_diff_srv_rate 0 dst_host_same_src_port_rate 0 dst_host_srv_diff_host_rate 0 dst_host_serror_rate 0 dst_host_srv_serror_rate 0 dst_host_rerror_rate 0 dst_host_srv_rerror_rate 0 labels 0 dtype: int64
Generates summary statistics for all numeric columns¶
In [24]:
df.describe()
Out[24]:
| duration | src_bytes | dst_bytes | land | wrong_fragment | urgent | hot | num_failed_logins | logged_in | num_compromised | ... | dst_host_count | dst_host_srv_count | dst_host_same_srv_rate | dst_host_diff_srv_rate | dst_host_same_src_port_rate | dst_host_srv_diff_host_rate | dst_host_serror_rate | dst_host_srv_serror_rate | dst_host_rerror_rate | dst_host_srv_rerror_rate | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 125973.00000 | 1.259730e+05 | 1.259730e+05 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | ... | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 | 125973.000000 |
| mean | 287.14465 | 4.556674e+04 | 1.977911e+04 | 0.000198 | 0.022687 | 0.000111 | 0.204409 | 0.001222 | 0.395736 | 0.279250 | ... | 182.148945 | 115.653005 | 0.521242 | 0.082951 | 0.148379 | 0.032542 | 0.284452 | 0.278485 | 0.118832 | 0.120240 |
| std | 2604.51531 | 5.870331e+06 | 4.021269e+06 | 0.014086 | 0.253530 | 0.014366 | 2.149968 | 0.045239 | 0.489010 | 23.942042 | ... | 99.206213 | 110.702741 | 0.448949 | 0.188922 | 0.308997 | 0.112564 | 0.444784 | 0.445669 | 0.306557 | 0.319459 |
| min | 0.00000 | 0.000000e+00 | 0.000000e+00 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ... | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| 25% | 0.00000 | 0.000000e+00 | 0.000000e+00 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ... | 82.000000 | 10.000000 | 0.050000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| 50% | 0.00000 | 4.400000e+01 | 0.000000e+00 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ... | 255.000000 | 63.000000 | 0.510000 | 0.020000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| 75% | 0.00000 | 2.760000e+02 | 5.160000e+02 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.000000 | 0.000000 | ... | 255.000000 | 255.000000 | 1.000000 | 0.070000 | 0.060000 | 0.020000 | 1.000000 | 1.000000 | 0.000000 | 0.000000 |
| max | 42908.00000 | 1.379964e+09 | 1.309937e+09 | 1.000000 | 3.000000 | 3.000000 | 77.000000 | 5.000000 | 1.000000 | 7479.000000 | ... | 255.000000 | 255.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
8 rows × 38 columns
Automatically identifying prediction target column¶
In [55]:
target = df.columns[-1]
Counts how many times each unique class appears¶
In [56]:
df[target].value_counts()
Out[56]:
labels normal 67343 neptune 41214 satan 3633 ipsweep 3599 portsweep 2931 smurf 2646 nmap 1493 back 956 teardrop 892 warezclient 890 pod 201 guess_passwd 53 buffer_overflow 30 warezmaster 20 land 18 imap 11 rootkit 10 loadmodule 9 ftp_write 8 multihop 7 phf 4 perl 3 spy 2 Name: count, dtype: int64
Visualizes how many samples exist per class¶
In [57]:
# Plot target distribution
plt.figure(figsize=(13,11))
sns.countplot(x=df[target])
plt.title("Attack Class Distribution")
plt.xticks(rotation=45)
plt.show()
Visualizing how strongly numeric features in the dataset¶
In [43]:
# Correlation heatmap (optional for large datasets)
plt.figure(figsize=(11,8))
sns.heatmap(df.corr(numeric_only=True).abs(), cmap="magma")
plt.title("Correlation Heatmap")
plt.show()
DATA PREPROCESSING¶
In [29]:
# Separate features & labels
X = df.drop(target, axis=1)
y = df[target]
In [30]:
# Identify categorical columns
categorical_cols = X.select_dtypes(include=["object"]).columns
In [31]:
# Label encoding
le = LabelEncoder()
for col in categorical_cols:
X[col] = le.fit_transform(X[col])
In [32]:
# Encode target if needed
y = LabelEncoder().fit_transform(y)
In [33]:
# Feature Scaling
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
In [34]:
# Train-Test Split
X_train, X_test, y_train, y_test = train_test_split(
X_scaled, y, test_size=0.25, random_state=42
)
Random Forest¶
- I will build the model using RandomForestClassifier
In [35]:
# Building Random Forest model
rf = RandomForestClassifier(n_estimators=100, random_state=42)
# Fitting Random Forest model
rf.fit(X_train, y_train)
rf_pred = rf.predict(X_test)
print("\nRandom Forest Accuracy:", accuracy_score(y_test, rf_pred))
print("\nClassification Report:\n", classification_report(y_test, rf_pred))
Random Forest Accuracy: 0.9982218835333715
Classification Report:
precision recall f1-score support
0 1.00 1.00 1.00 222
1 0.67 0.40 0.50 10
3 1.00 0.92 0.96 13
4 1.00 0.67 0.80 3
5 0.99 0.99 0.99 918
6 0.33 0.25 0.29 4
7 0.00 0.00 0.00 1
9 1.00 1.00 1.00 10316
10 0.99 0.99 0.99 383
11 1.00 1.00 1.00 16774
12 1.00 1.00 1.00 1
13 1.00 1.00 1.00 1
14 1.00 0.94 0.97 53
15 1.00 1.00 1.00 727
16 0.00 0.00 0.00 3
17 1.00 0.99 0.99 919
18 1.00 1.00 1.00 662
19 0.00 0.00 0.00 1
20 1.00 1.00 1.00 227
21 1.00 0.98 0.99 248
22 0.88 0.88 0.88 8
accuracy 1.00 31494
macro avg 0.80 0.76 0.78 31494
weighted avg 1.00 1.00 1.00 31494
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\metrics\_classification.py:1565: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\metrics\_classification.py:1565: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\metrics\_classification.py:1565: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
Creating a confusion matrix for Random Forest predictions¶
In [60]:
# Generates a confusion matrix for the Random Forest predictions.
cm_rf = confusion_matrix(y_test, rf_pred)
plt.figure(figsize=(13,8))
# Displays the confusion matrix as a color-coded grid.
sns.heatmap(cm_rf, annot=True, fmt="d", cmap="magma")
plt.title("Random Forest - Confusion Matrix")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()
Gradient Boosting¶
- I will build the model using GradientBoostingClassifier
In [44]:
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
import seaborn as sns
import matplotlib.pyplot as plt
print("\nTraining Gradient Boosting...")
# Faster version
gb = GradientBoostingClassifier(
n_estimators=60, # default 100, reduced for speed
max_depth=2, # shallow trees, faster
learning_rate=0.1,
random_state=42
)
gb.fit(X_train, y_train)
gb_pred = gb.predict(X_test)
print("\nGradient Boosting Accuracy:", accuracy_score(y_test, gb_pred))
print("\nClassification Report:\n", classification_report(y_test, gb_pred))
Training Gradient Boosting...
Gradient Boosting Accuracy: 0.9918079634216042
Classification Report:
precision recall f1-score support
0 0.91 0.99 0.95 222
1 0.03 0.20 0.05 10
2 0.00 0.00 0.00 0
3 0.00 0.00 0.00 13
4 0.50 0.67 0.57 3
5 0.99 0.98 0.98 918
6 0.67 1.00 0.80 4
7 0.00 0.00 0.00 1
9 1.00 1.00 1.00 10316
10 0.97 0.91 0.94 383
11 0.99 0.99 0.99 16774
12 0.00 0.00 0.00 1
13 0.33 1.00 0.50 1
14 1.00 0.94 0.97 53
15 0.99 0.98 0.99 727
16 0.00 0.00 0.00 3
17 1.00 0.97 0.99 919
18 1.00 1.00 1.00 662
19 0.00 0.00 0.00 1
20 1.00 1.00 1.00 227
21 1.00 0.95 0.97 248
22 0.86 0.75 0.80 8
accuracy 0.99 31494
macro avg 0.60 0.65 0.61 31494
weighted avg 0.99 0.99 0.99 31494
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\metrics\_classification.py:1565: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\metrics\_classification.py:1565: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\metrics\_classification.py:1565: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\metrics\_classification.py:1565: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\metrics\_classification.py:1565: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\metrics\_classification.py:1565: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
Creating a confusion matrix for Gradient Boosting predictions¶
In [45]:
# Generates a confusion matrix for the Gradient Boosting predictions.
cm = confusion_matrix(y_test, gb_pred)
plt.figure(figsize=(13,8))
# Displays the confusion matrix as a color-coded grid.
sns.heatmap(cm, annot=True, fmt="d", cmap="magma")
plt.title("Gradient Boosting - Confusion Matrix (Fast Version)")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()
Hyperparameter Tuning¶
In [50]:
# Prints a message to indicate that hyperparameter tuning
print("\nPerforming Hyperparameter Tuning for Random Forest...")
param_grid_rf = {
"n_estimators": [100, 150],
"max_depth": [10, 20, None],
"min_samples_split": [2, 5],
}
grid_rf = GridSearchCV(
RandomForestClassifier(random_state=42),
param_grid_rf,
cv=3,
n_jobs=-1,
verbose=1
)
# Trains the GridSearch model on the training data.
grid_rf.fit(X_train, y_train)
# Prints the best hyperparameters found by GridSearchCV.
print("\nBest RF Parameters:", grid_rf.best_params_)
# Extracts the best Random Forest model and makes predictions on the test set.
rf_best = grid_rf.best_estimator_
rf_best_pred = rf_best.predict(X_test)
print("\nTuned RF Accuracy:", accuracy_score(y_test, rf_best_pred))
print("\nConfusion Matrix (Tuned RF):")
print(confusion_matrix(y_test, rf_best_pred))
Performing Hyperparameter Tuning for Random Forest... Fitting 3 folds for each of 12 candidates, totalling 36 fits
C:\Users\erhan\anaconda3\Lib\site-packages\sklearn\model_selection\_split.py:805: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=3. warnings.warn(
Best RF Parameters: {'max_depth': None, 'min_samples_split': 2, 'n_estimators': 150}
Tuned RF Accuracy: 0.9981266272940877
Confusion Matrix (Tuned RF):
[[ 222 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0]
[ 0 4 0 0 0 0 0 0 0 5 0 0
0 0 0 0 0 0 0 0 1]
[ 0 0 12 0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 0 0 0]
[ 0 0 0 2 0 0 0 0 0 1 0 0
0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 913 0 0 0 4 1 0 0
0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 1 0 0 0 2 0 0
0 1 0 0 0 0 0 0 0]
[ 0 1 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 10314 0 2 0 0
0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 5 0 0 0 378 0 0 0
0 0 0 0 0 0 0 0 0]
[ 0 1 0 0 0 2 0 0 1 16766 0 0
0 0 0 3 0 0 0 1 0]
[ 0 0 0 0 0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 1
0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 3 0 0
50 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 3 0 0
0 724 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 3 0 0
0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 1 0 0 0 0 8 0 0
0 1 0 909 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 662 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 227 0 0]
[ 0 0 0 0 0 0 0 0 0 5 0 0
0 0 0 0 0 0 0 243 0]
[ 0 0 0 0 0 0 0 0 0 2 0 0
0 0 0 0 0 0 0 0 6]]
Observations:¶
- Tuning Random Forest with GridSearchCV improved performance slightly.
- Indicates that careful selection of hyperparameters can enhance accuracy and generalization.
Model Comparison¶
In [51]:
# Prepare results
model_names = ["Random Forest", "Gradient Boosting", "Random Forest (Tuned)"]
accuracies = [
accuracy_score(y_test, rf_pred),
accuracy_score(y_test, gb_pred),
accuracy_score(y_test, rf_best_pred)
]
# Convert to DataFrame
comparison_df = pd.DataFrame({
"Model": model_names,
"Accuracy": accuracies
})
# Visualization
plt.figure(figsize=(13,11))
sns.barplot(x="Model", y="Accuracy", data=comparison_df)
plt.title("Final Model Accuracy Comparison")
plt.ylim(0, 1)
plt.ylabel("Accuracy Score")
plt.xlabel("Machine Learning Models")
plt.xticks(rotation=15)
plt.show()
# Automatically displays the table
comparison_df
Out[51]:
| Model | Accuracy | |
|---|---|---|
| 0 | Random Forest | 0.998222 |
| 1 | Gradient Boosting | 0.991808 |
| 2 | Random Forest (Tuned) | 0.998127 |
Summary¶
In [48]:
import pandas as pd
from sklearn.metrics import accuracy_score
# Collect all model accuracies
summary_data = {
"Model": ["Random Forest", "Gradient Boosting", "Random Forest (Tuned)"],
"Accuracy": [
accuracy_score(y_test, rf_pred),
accuracy_score(y_test, gb_pred),
accuracy_score(y_test, rf_best_pred)
]
}
# Create DataFrame
summary_df = pd.DataFrame(summary_data)
# Identify the best model
best_model = summary_df.loc[summary_df["Accuracy"].idxmax()]
summary_df, best_model
Out[48]:
( Model Accuracy 0 Random Forest 0.998222 1 Gradient Boosting 0.991808 2 Random Forest (Tuned) 0.998127, Model Random Forest Accuracy 0.998222 Name: 0, dtype: object)
Observations:¶
- Random Forest model achieved a slightly higher accuracy (0.998222) compared to the tuned model (0.998127).