NSL-KDD Intrusion Detection: Ensemble Learning¶
Multi-class Network Intrusion Classification using Random Forest and Gradient Boosting¶
This notebook develops and evaluates two ensemble machine learning models for multi-class classification of network intrusions using the NSL-KDD dataset. The pipeline covers exploratory data analysis, preprocessing, model training with hyperparameter tuning, evaluation, and feature importance analysis.
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import MinMaxScaler, LabelEncoder
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import (classification_report, confusion_matrix,
accuracy_score, f1_score)
1. Data Loading¶
In [2]:
train = pd.read_csv("kdd_train (1).csv")
test = pd.read_csv("kdd_test (2).csv")
print("train shape:", train.shape)
print("test shape: ", test.shape)
train shape: (125973, 42) test shape: (22544, 42)
In [3]:
columns = [
'duration', 'protocol_type', 'service', 'flag', 'src_bytes', 'dst_bytes',
'land', 'wrong_fragment', 'urgent', 'hot', 'num_failed_logins', 'logged_in',
'num_compromised', 'root_shell', 'su_attempted', 'num_root',
'num_file_creations', 'num_shells', 'num_access_files', 'num_outbound_cmds',
'is_host_login', 'is_guest_login', 'count', 'srv_count', 'serror_rate',
'srv_serror_rate', 'rerror_rate', 'srv_rerror_rate', 'same_srv_rate',
'diff_srv_rate', 'srv_diff_host_rate', 'dst_host_count', 'dst_host_srv_count',
'dst_host_same_srv_rate', 'dst_host_diff_srv_rate',
'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate',
'dst_host_serror_rate', 'dst_host_srv_serror_rate', 'dst_host_rerror_rate',
'dst_host_srv_rerror_rate', 'label'
]
train.columns = columns
test.columns = columns
print("columns assigned successfully")
print("\nsample labels:")
print(train['label'].value_counts().head(10))
columns assigned successfully sample labels: label normal 67343 neptune 41214 satan 3633 ipsweep 3599 portsweep 2931 smurf 2646 nmap 1493 back 956 teardrop 892 warezclient 890 Name: count, dtype: int64
In [4]:
dos_attacks = ['back', 'land', 'neptune', 'pod', 'smurf', 'teardrop',
'mailbomb', 'apache2', 'processtable', 'udpstorm']
probe_attacks = ['satan', 'ipsweep', 'nmap', 'portsweep', 'mscan', 'saint']
r2l_attacks = ['ftp_write', 'guess_passwd', 'imap', 'multihop', 'phf', 'spy',
'warezclient', 'warezmaster', 'sendmail', 'named']
u2r_attacks = ['buffer_overflow', 'loadmodule', 'rootkit', 'perl',
'sqlattack', 'xterm', 'ps']
def map_label(label):
if label == 'normal':
return 'normal'
elif label in dos_attacks:
return 'dos'
elif label in probe_attacks:
return 'probe'
elif label in r2l_attacks:
return 'r2l'
elif label in u2r_attacks:
return 'u2r'
return None
train['label_mc'] = train['label'].apply(map_label)
test['label_mc'] = test['label'].apply(map_label)
# remove any test rows whose label did not match a known category
test = test.dropna(subset=['label_mc'])
# would remove duplicates at this stage
# not necessary for NSL-KDD
print("training class distribution:")
print(train['label_mc'].value_counts())
print("\ntest class distribution:")
print(test['label_mc'].value_counts())
training class distribution: label_mc normal 67343 dos 45927 probe 11656 r2l 995 u2r 52 Name: count, dtype: int64 test class distribution: label_mc normal 11245 dos 8095 probe 2157 r2l 821 u2r 38 Name: count, dtype: int64
2. Exploratory Data Analysis¶
The following section examines the dataset from multiple angles before any modelling takes place. Class distribution, feature distributions, correlations, and categorical breakdowns are all investigated to inform preprocessing and modelling decisions.
In [5]:
print("training set info:")
print(f" rows: {train.shape[0]}, columns: {train.shape[1]}")
print(f"\nmissing values in training set: {train.isnull().sum().sum()}")
print(f"missing values in test set: {test.isnull().sum().sum()}")
print(f"\ntraining set dtypes:\n{train.dtypes.value_counts()}")
print(f"\nnumeric feature summary:")
display(train.describe().T.round(3))
training set info: rows: 125973, columns: 43 missing values in training set: 0 missing values in test set: 0 training set dtypes: int64 23 float64 15 object 5 Name: count, dtype: int64 numeric feature summary:
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| duration | 125973.0 | 287.145 | 2604.515 | 0.0 | 0.00 | 0.00 | 0.00 | 4.290800e+04 |
| src_bytes | 125973.0 | 45566.743 | 5870331.182 | 0.0 | 0.00 | 44.00 | 276.00 | 1.379964e+09 |
| dst_bytes | 125973.0 | 19779.114 | 4021269.151 | 0.0 | 0.00 | 0.00 | 516.00 | 1.309937e+09 |
| land | 125973.0 | 0.000 | 0.014 | 0.0 | 0.00 | 0.00 | 0.00 | 1.000000e+00 |
| wrong_fragment | 125973.0 | 0.023 | 0.254 | 0.0 | 0.00 | 0.00 | 0.00 | 3.000000e+00 |
| urgent | 125973.0 | 0.000 | 0.014 | 0.0 | 0.00 | 0.00 | 0.00 | 3.000000e+00 |
| hot | 125973.0 | 0.204 | 2.150 | 0.0 | 0.00 | 0.00 | 0.00 | 7.700000e+01 |
| num_failed_logins | 125973.0 | 0.001 | 0.045 | 0.0 | 0.00 | 0.00 | 0.00 | 5.000000e+00 |
| logged_in | 125973.0 | 0.396 | 0.489 | 0.0 | 0.00 | 0.00 | 1.00 | 1.000000e+00 |
| num_compromised | 125973.0 | 0.279 | 23.942 | 0.0 | 0.00 | 0.00 | 0.00 | 7.479000e+03 |
| root_shell | 125973.0 | 0.001 | 0.037 | 0.0 | 0.00 | 0.00 | 0.00 | 1.000000e+00 |
| su_attempted | 125973.0 | 0.001 | 0.045 | 0.0 | 0.00 | 0.00 | 0.00 | 2.000000e+00 |
| num_root | 125973.0 | 0.302 | 24.400 | 0.0 | 0.00 | 0.00 | 0.00 | 7.468000e+03 |
| num_file_creations | 125973.0 | 0.013 | 0.484 | 0.0 | 0.00 | 0.00 | 0.00 | 4.300000e+01 |
| num_shells | 125973.0 | 0.000 | 0.022 | 0.0 | 0.00 | 0.00 | 0.00 | 2.000000e+00 |
| num_access_files | 125973.0 | 0.004 | 0.099 | 0.0 | 0.00 | 0.00 | 0.00 | 9.000000e+00 |
| num_outbound_cmds | 125973.0 | 0.000 | 0.000 | 0.0 | 0.00 | 0.00 | 0.00 | 0.000000e+00 |
| is_host_login | 125973.0 | 0.000 | 0.003 | 0.0 | 0.00 | 0.00 | 0.00 | 1.000000e+00 |
| is_guest_login | 125973.0 | 0.009 | 0.097 | 0.0 | 0.00 | 0.00 | 0.00 | 1.000000e+00 |
| count | 125973.0 | 84.108 | 114.509 | 0.0 | 2.00 | 14.00 | 143.00 | 5.110000e+02 |
| srv_count | 125973.0 | 27.738 | 72.636 | 0.0 | 2.00 | 8.00 | 18.00 | 5.110000e+02 |
| serror_rate | 125973.0 | 0.284 | 0.446 | 0.0 | 0.00 | 0.00 | 1.00 | 1.000000e+00 |
| srv_serror_rate | 125973.0 | 0.282 | 0.447 | 0.0 | 0.00 | 0.00 | 1.00 | 1.000000e+00 |
| rerror_rate | 125973.0 | 0.120 | 0.320 | 0.0 | 0.00 | 0.00 | 0.00 | 1.000000e+00 |
| srv_rerror_rate | 125973.0 | 0.121 | 0.324 | 0.0 | 0.00 | 0.00 | 0.00 | 1.000000e+00 |
| same_srv_rate | 125973.0 | 0.661 | 0.440 | 0.0 | 0.09 | 1.00 | 1.00 | 1.000000e+00 |
| diff_srv_rate | 125973.0 | 0.063 | 0.180 | 0.0 | 0.00 | 0.00 | 0.06 | 1.000000e+00 |
| srv_diff_host_rate | 125973.0 | 0.097 | 0.260 | 0.0 | 0.00 | 0.00 | 0.00 | 1.000000e+00 |
| dst_host_count | 125973.0 | 182.149 | 99.206 | 0.0 | 82.00 | 255.00 | 255.00 | 2.550000e+02 |
| dst_host_srv_count | 125973.0 | 115.653 | 110.703 | 0.0 | 10.00 | 63.00 | 255.00 | 2.550000e+02 |
| dst_host_same_srv_rate | 125973.0 | 0.521 | 0.449 | 0.0 | 0.05 | 0.51 | 1.00 | 1.000000e+00 |
| dst_host_diff_srv_rate | 125973.0 | 0.083 | 0.189 | 0.0 | 0.00 | 0.02 | 0.07 | 1.000000e+00 |
| dst_host_same_src_port_rate | 125973.0 | 0.148 | 0.309 | 0.0 | 0.00 | 0.00 | 0.06 | 1.000000e+00 |
| dst_host_srv_diff_host_rate | 125973.0 | 0.033 | 0.113 | 0.0 | 0.00 | 0.00 | 0.02 | 1.000000e+00 |
| dst_host_serror_rate | 125973.0 | 0.284 | 0.445 | 0.0 | 0.00 | 0.00 | 1.00 | 1.000000e+00 |
| dst_host_srv_serror_rate | 125973.0 | 0.278 | 0.446 | 0.0 | 0.00 | 0.00 | 1.00 | 1.000000e+00 |
| dst_host_rerror_rate | 125973.0 | 0.119 | 0.307 | 0.0 | 0.00 | 0.00 | 0.00 | 1.000000e+00 |
| dst_host_srv_rerror_rate | 125973.0 | 0.120 | 0.319 | 0.0 | 0.00 | 0.00 | 0.00 | 1.000000e+00 |
In [6]:
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
train_counts = train['label_mc'].value_counts()
test_counts = test['label_mc'].value_counts()
for ax, counts, title, colour in zip(
axes,
[train_counts, test_counts],
['training set class distribution', 'test set class distribution'],
['steelblue', 'coral']
):
bars = ax.bar(counts.index, counts.values, color=colour)
ax.set_title(title, fontsize=12)
ax.set_xlabel('class')
ax.set_ylabel('count')
for bar, val in zip(bars, counts.values):
ax.text(bar.get_x() + bar.get_width() / 2,
val + counts.max() * 0.01,
f'{val:,}', ha='center', fontsize=9)
plt.suptitle('class distribution: training vs test', fontsize=13)
plt.tight_layout()
plt.savefig('class_distribution.png', dpi=150)
plt.show()
balance_ratio = train_counts.min() / train_counts.max()
print(f"training balance ratio (min/max): {balance_ratio:.4f}")
print("dataset is heavily imbalanced — macro F1 is the appropriate primary metric")
training balance ratio (min/max): 0.0008 dataset is heavily imbalanced — macro F1 is the appropriate primary metric
In [7]:
proportions = pd.DataFrame({
'train_count': train_counts,
'train_pct': (train_counts / train_counts.sum() * 100).round(2),
'test_count': test_counts,
'test_pct': (test_counts / test_counts.sum() * 100).round(2)
})
print(proportions)
train_count train_pct test_count test_pct label_mc normal 67343 53.46 11245 50.30 dos 45927 36.46 8095 36.21 probe 11656 9.25 2157 9.65 r2l 995 0.79 821 3.67 u2r 52 0.04 38 0.17
In [8]:
cat_cols = ['protocol_type', 'service', 'flag']
for col in cat_cols:
print(f"\n{'='*55}")
print(f" {col} breakdown by attack class")
print('='*55)
ct = pd.crosstab(train[col], train['label_mc'], normalize='columns').round(3)
display(ct)
======================================================= protocol_type breakdown by attack class =======================================================
| label_mc | dos | normal | probe | r2l | u2r |
|---|---|---|---|---|---|
| protocol_type | |||||
| icmp | 0.062 | 0.019 | 0.355 | 0.0 | 0.000 |
| tcp | 0.919 | 0.796 | 0.502 | 1.0 | 0.942 |
| udp | 0.019 | 0.185 | 0.143 | 0.0 | 0.058 |
======================================================= service breakdown by attack class =======================================================
| label_mc | dos | normal | probe | r2l | u2r |
|---|---|---|---|---|---|
| service | |||||
| IRC | 0.000 | 0.003 | 0.000 | 0.0 | 0.0 |
| X11 | 0.000 | 0.001 | 0.001 | 0.0 | 0.0 |
| Z39_50 | 0.019 | 0.000 | 0.001 | 0.0 | 0.0 |
| aol | 0.000 | 0.000 | 0.000 | 0.0 | 0.0 |
| auth | 0.015 | 0.004 | 0.001 | 0.0 | 0.0 |
| ... | ... | ... | ... | ... | ... |
| urp_i | 0.000 | 0.009 | 0.000 | 0.0 | 0.0 |
| uucp | 0.017 | 0.000 | 0.001 | 0.0 | 0.0 |
| uucp_path | 0.015 | 0.000 | 0.001 | 0.0 | 0.0 |
| vmnet | 0.013 | 0.000 | 0.001 | 0.0 | 0.0 |
| whois | 0.015 | 0.000 | 0.002 | 0.0 | 0.0 |
70 rows × 5 columns
======================================================= flag breakdown by attack class =======================================================
| label_mc | dos | normal | probe | r2l | u2r |
|---|---|---|---|---|---|
| flag | |||||
| OTH | 0.000 | 0.000 | 0.003 | 0.000 | 0.000 |
| REJ | 0.123 | 0.040 | 0.246 | 0.000 | 0.000 |
| RSTO | 0.026 | 0.003 | 0.007 | 0.046 | 0.019 |
| RSTOS0 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 |
| RSTR | 0.002 | 0.002 | 0.187 | 0.005 | 0.000 |
| S0 | 0.748 | 0.005 | 0.013 | 0.000 | 0.000 |
| S1 | 0.000 | 0.005 | 0.000 | 0.001 | 0.000 |
| S2 | 0.000 | 0.002 | 0.000 | 0.001 | 0.000 |
| S3 | 0.000 | 0.001 | 0.000 | 0.003 | 0.000 |
| SF | 0.100 | 0.941 | 0.512 | 0.940 | 0.981 |
| SH | 0.000 | 0.000 | 0.023 | 0.004 | 0.000 |
In [9]:
fig, axes = plt.subplots(1, 3, figsize=(16, 5))
for ax, col in zip(axes, cat_cols):
ct = pd.crosstab(train[col], train['label_mc'])
ct.plot(kind='bar', ax=ax, stacked=True, colormap='Set2', legend=(col == 'flag'))
ax.set_title(f'{col} by attack class')
ax.set_xlabel(col)
ax.set_ylabel('count')
ax.tick_params(axis='x', rotation=45)
plt.suptitle('categorical feature distributions by attack class', fontsize=13)
plt.tight_layout()
plt.savefig('categorical_eda.png', dpi=150)
plt.show()
In [10]:
# select features known to be informative for intrusion detection
key_features = ['duration', 'src_bytes', 'dst_bytes', 'count',
'srv_count', 'same_srv_rate', 'serror_rate', 'dst_host_count']
classes = train['label_mc'].unique()
colours = ['steelblue', 'coral', 'seagreen', 'orchid', 'goldenrod']
class_colour = dict(zip(sorted(classes), colours))
fig, axes = plt.subplots(2, 4, figsize=(18, 8))
axes = axes.flatten()
for i, feat in enumerate(key_features):
for cls in sorted(classes):
vals = train[train['label_mc'] == cls][feat]
# log1p to handle zero-heavy, highly skewed distributions
axes[i].hist(np.log1p(vals), bins=40, alpha=0.5,
label=cls, color=class_colour[cls], density=True)
axes[i].set_title(f'{feat} (log scale)')
axes[i].set_xlabel('log(1 + value)')
axes[i].set_ylabel('density')
if i == 0:
axes[i].legend(fontsize=7)
plt.suptitle('numeric feature distributions by class (log-transformed)', fontsize=13)
plt.tight_layout()
plt.savefig('feature_distributions.png', dpi=150)
plt.show()
In [11]:
numeric_cols = train.select_dtypes(include=[np.number]).columns.tolist()
corr = train[numeric_cols].corr()
plt.figure(figsize=(16, 13))
mask = np.triu(np.ones_like(corr, dtype=bool)) # show lower triangle only
sns.heatmap(corr, mask=mask, cmap='coolwarm', center=0,
linewidths=0.2, vmin=-1, vmax=1,
xticklabels=False, yticklabels=False)
plt.title('feature correlation matrix (lower triangle)', fontsize=13)
plt.tight_layout()
plt.savefig('correlation_heatmap.png', dpi=150)
plt.show()
# identify highly correlated pairs (|r| > 0.9) for discussion
corr_pairs = (corr.abs()
.where(np.tril(np.ones(corr.shape), k=-1).astype(bool))
.stack()
.sort_values(ascending=False))
print("top 10 most correlated feature pairs:")
print(corr_pairs.head(10).round(3))
top 10 most correlated feature pairs:
num_root num_compromised 0.999
srv_serror_rate serror_rate 0.993
srv_rerror_rate rerror_rate 0.989
dst_host_srv_serror_rate srv_serror_rate 0.986
dst_host_serror_rate 0.985
serror_rate 0.981
dst_host_serror_rate serror_rate 0.979
srv_serror_rate 0.978
dst_host_srv_rerror_rate srv_rerror_rate 0.970
rerror_rate 0.964
dtype: float64
3. Preprocessing¶
Preprocessing covers three steps: feature/label separation, label encoding of categorical columns, and MinMax normalisation of all features. All transformations are fitted on training data only and applied to the test set, preventing data leakage.
In [12]:
X_train_raw = train.drop(['label', 'label_mc'], axis=1)
y_train = train['label_mc']
X_test_raw = test.drop(['label', 'label_mc'], axis=1)
y_test = test['label_mc']
print("X_train:", X_train_raw.shape, " y_train:", y_train.shape)
print("X_test: ", X_test_raw.shape, " y_test: ", y_test.shape)
X_train: (125973, 41) y_train: (125973,) X_test: (22356, 41) y_test: (22356,)
In [13]:
cat_cols = ['protocol_type', 'service', 'flag']
le_dict = {}
X_train_enc = X_train_raw.copy()
X_test_enc = X_test_raw.copy()
for col in cat_cols:
le = LabelEncoder()
X_train_enc[col] = le.fit_transform(X_train_enc[col])
# transform only — encoder is fit on training data exclusively
X_test_enc[col] = le.transform(X_test_enc[col])
le_dict[col] = le
print(f"{col}: {len(le.classes_)} categories encoded")
protocol_type: 3 categories encoded service: 70 categories encoded flag: 11 categories encoded
In [14]:
scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train_enc)
X_test_scaled = scaler.transform(X_test_enc)
print("scaling complete")
print(f"feature value range after scaling: [{X_train_scaled.min():.2f}, {X_train_scaled.max():.2f}]")
print(f"X_train_scaled shape: {X_train_scaled.shape}")
print(f"X_test_scaled shape: {X_test_scaled.shape}")
scaling complete feature value range after scaling: [0.00, 1.00] X_train_scaled shape: (125973, 41) X_test_scaled shape: (22356, 41)
4. Random Forest Classifier¶
Random Forest is a bagging ensemble that trains multiple decision trees on random subsets of the training data and features, aggregating their predictions by majority vote. It handles high-dimensional data well and is robust to overfitting through the decorrelation introduced by feature sub-sampling at each split.
4.1 Hyperparameter Tuning¶
GridSearchCV with 3-fold cross-validation is used to identify the optimal combination of n_estimators and max_depth. The parameter grid is deliberately scoped to balance search thoroughness against computational cost, a necessary consideration given the size of the NSL-KDD training set (125,973 samples). Macro-averaged F1 is used as the optimisation criterion to account for class imbalance.
In [15]:
rf_param_grid = {
'n_estimators': [100, 200],
'max_depth': [10, 20, None],
}
rf_base = RandomForestClassifier(random_state=42, n_jobs=-1)
rf_grid = GridSearchCV(
estimator = rf_base,
param_grid = rf_param_grid,
cv = 3,
scoring = 'f1_macro',
n_jobs = -1,
verbose = 1
)
rf_grid.fit(X_train_scaled, y_train)
print("\nbest RF parameters:", rf_grid.best_params_)
print("best CV macro F1: ", round(rf_grid.best_score_, 4))
Fitting 3 folds for each of 6 candidates, totalling 18 fits
best RF parameters: {'max_depth': None, 'n_estimators': 100}
best CV macro F1: 0.9262
In [16]:
rf_cv_results = pd.DataFrame(rf_grid.cv_results_)[[
'param_n_estimators', 'param_max_depth',
'mean_test_score', 'std_test_score', 'rank_test_score'
]]
rf_cv_results.columns = ['n_estimators', 'max_depth',
'mean_f1_macro', 'std_f1_macro', 'rank']
rf_cv_results = rf_cv_results.sort_values('rank')
display(rf_cv_results.style.format({
'mean_f1_macro': '{:.4f}',
'std_f1_macro': '{:.4f}'
}))
| n_estimators | max_depth | mean_f1_macro | std_f1_macro | rank | |
|---|---|---|---|---|---|
| 4 | 100 | None | 0.9262 | 0.0011 | 1 |
| 5 | 200 | None | 0.9226 | 0.0030 | 2 |
| 2 | 100 | 20 | 0.9201 | 0.0040 | 3 |
| 3 | 200 | 20 | 0.9188 | 0.0100 | 4 |
| 0 | 100 | 10 | 0.8734 | 0.0136 | 5 |
| 1 | 200 | 10 | 0.8682 | 0.0109 | 6 |
4.2 Evaluation of Best Random Forest Model¶
In [17]:
best_rf = rf_grid.best_estimator_
y_pred_rf = best_rf.predict(X_test_scaled)
rf_acc = accuracy_score(y_test, y_pred_rf)
rf_f1 = f1_score(y_test, y_pred_rf, average='macro')
print(f"best parameters : {rf_grid.best_params_}")
print(f"accuracy : {rf_acc:.4f}")
print(f"macro F1 : {rf_f1:.4f}")
print()
print(classification_report(y_test, y_pred_rf))
best parameters : {'max_depth': None, 'n_estimators': 100}
accuracy : 0.9300
macro F1 : 0.7233
precision recall f1-score support
dos 0.99 0.93 0.96 8095
normal 0.89 0.99 0.94 11245
probe 0.96 0.88 0.92 2157
r2l 0.99 0.19 0.32 821
u2r 1.00 0.32 0.48 38
accuracy 0.93 22356
macro avg 0.97 0.66 0.72 22356
weighted avg 0.94 0.93 0.92 22356
In [18]:
cm_rf = confusion_matrix(y_test, y_pred_rf, labels=best_rf.classes_)
cm_rf_norm = cm_rf.astype('float') / cm_rf.sum(axis=1)[:, np.newaxis]
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
sns.heatmap(cm_rf, annot=True, fmt='d', cmap='Blues',
xticklabels=best_rf.classes_,
yticklabels=best_rf.classes_, ax=axes[0])
axes[0].set_title('random forest — raw counts')
axes[0].set_xlabel('predicted')
axes[0].set_ylabel('actual')
sns.heatmap(cm_rf_norm, annot=True, fmt='.2f', cmap='Blues',
xticklabels=best_rf.classes_,
yticklabels=best_rf.classes_, ax=axes[1])
axes[1].set_title('random forest — normalised')
axes[1].set_xlabel('predicted')
axes[1].set_ylabel('actual')
plt.suptitle('random forest confusion matrices', fontsize=13)
plt.tight_layout()
plt.savefig('rf_confusion_matrices.png', dpi=150)
plt.show()
In [19]:
fi_rf = pd.DataFrame({
'feature': X_train_enc.columns,
'importance': best_rf.feature_importances_
}).sort_values('importance', ascending=False).head(15)
plt.figure(figsize=(10, 6))
bars = plt.barh(fi_rf['feature'], fi_rf['importance'], color='steelblue')
plt.gca().invert_yaxis()
plt.xlabel('importance score')
plt.title('top 15 features — random forest')
plt.tight_layout()
plt.savefig('rf_feature_importance.png', dpi=150)
plt.show()
print("\ntop 15 features:")
print(fi_rf.to_string(index=False))
top 15 features:
feature importance
src_bytes 0.156298
same_srv_rate 0.092124
flag 0.072725
dst_host_serror_rate 0.066743
dst_bytes 0.060896
count 0.052552
srv_serror_rate 0.046220
diff_srv_rate 0.046056
dst_host_same_srv_rate 0.040114
serror_rate 0.038470
dst_host_same_src_port_rate 0.036660
dst_host_diff_srv_rate 0.033698
protocol_type 0.029418
dst_host_srv_count 0.026435
dst_host_srv_serror_rate 0.026196
5. Gradient Boosting Classifier¶
Gradient Boosting is a sequential ensemble method that builds trees one at a time, with each tree correcting the residual errors of its predecessor. Unlike Random Forest, which parallelises tree construction, Gradient Boosting is inherently sequential, making it more computationally expensive but often more accurate on structured data, particularly for minority classes where precise decision boundaries matter.
5.1 Hyperparameter Tuning¶
The parameter grid covers n_estimators, learning_rate, and max_depth. The grid is scoped to a practical range given the training set size: larger grids would improve the search but at a computational cost disproportionate to the likely performance gain, a trade-off that is itself a relevant consideration when evaluating the suitability of Gradient Boosting for resource-constrained deployments.
In [20]:
gb_param_grid = {
'n_estimators': [100, 150],
'learning_rate': [0.05, 0.1],
'max_depth': [2, 3],
}
gb_base = GradientBoostingClassifier(random_state=42)
gb_grid = GridSearchCV(
estimator = gb_base,
param_grid = gb_param_grid,
cv = 3,
scoring = 'f1_macro',
n_jobs = -1,
verbose = 1
)
gb_grid.fit(X_train_scaled, y_train)
print("\nbest GB parameters:", gb_grid.best_params_)
print("best CV macro F1: ", round(gb_grid.best_score_, 4))
Fitting 3 folds for each of 8 candidates, totalling 24 fits
best GB parameters: {'learning_rate': 0.1, 'max_depth': 2, 'n_estimators': 150}
best CV macro F1: 0.8737
In [21]:
gb_cv_results = pd.DataFrame(gb_grid.cv_results_)[[
'param_n_estimators', 'param_learning_rate', 'param_max_depth',
'mean_test_score', 'std_test_score', 'rank_test_score'
]]
gb_cv_results.columns = ['n_estimators', 'learning_rate', 'max_depth',
'mean_f1_macro', 'std_f1_macro', 'rank']
gb_cv_results = gb_cv_results.sort_values('rank')
display(gb_cv_results.style.format({
'mean_f1_macro': '{:.4f}',
'std_f1_macro': '{:.4f}'
}))
| n_estimators | learning_rate | max_depth | mean_f1_macro | std_f1_macro | rank | |
|---|---|---|---|---|---|---|
| 5 | 150 | 0.100000 | 2 | 0.8737 | 0.0112 | 1 |
| 4 | 100 | 0.100000 | 2 | 0.8711 | 0.0227 | 2 |
| 1 | 150 | 0.050000 | 2 | 0.8695 | 0.0285 | 3 |
| 0 | 100 | 0.050000 | 2 | 0.8517 | 0.0278 | 4 |
| 7 | 150 | 0.100000 | 3 | 0.8498 | 0.0385 | 5 |
| 6 | 100 | 0.100000 | 3 | 0.8487 | 0.0380 | 6 |
| 2 | 100 | 0.050000 | 3 | 0.8301 | 0.0383 | 7 |
| 3 | 150 | 0.050000 | 3 | 0.8163 | 0.0127 | 8 |
5.2 Evaluation of Best Gradient Boosting Model¶
In [22]:
best_gb = gb_grid.best_estimator_
y_pred_gb = best_gb.predict(X_test_scaled)
gb_acc = accuracy_score(y_test, y_pred_gb)
gb_f1 = f1_score(y_test, y_pred_gb, average='macro')
print(f"best parameters : {gb_grid.best_params_}")
print(f"accuracy : {gb_acc:.4f}")
print(f"macro F1 : {gb_f1:.4f}")
print()
print(classification_report(y_test, y_pred_gb))
best parameters : {'learning_rate': 0.1, 'max_depth': 2, 'n_estimators': 150}
accuracy : 0.9306
macro F1 : 0.7163
precision recall f1-score support
dos 0.99 0.94 0.96 8095
normal 0.89 0.99 0.94 11245
probe 0.94 0.87 0.90 2157
r2l 0.98 0.19 0.32 821
u2r 0.80 0.32 0.45 38
accuracy 0.93 22356
macro avg 0.92 0.66 0.72 22356
weighted avg 0.94 0.93 0.92 22356
In [23]:
cm_gb = confusion_matrix(y_test, y_pred_gb, labels=best_gb.classes_)
cm_gb_norm = cm_gb.astype('float') / cm_gb.sum(axis=1)[:, np.newaxis]
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
sns.heatmap(cm_gb, annot=True, fmt='d', cmap='Greens',
xticklabels=best_gb.classes_,
yticklabels=best_gb.classes_, ax=axes[0])
axes[0].set_title('gradient boosting — raw counts')
axes[0].set_xlabel('predicted')
axes[0].set_ylabel('actual')
sns.heatmap(cm_gb_norm, annot=True, fmt='.2f', cmap='Greens',
xticklabels=best_gb.classes_,
yticklabels=best_gb.classes_, ax=axes[1])
axes[1].set_title('gradient boosting — normalised')
axes[1].set_xlabel('predicted')
axes[1].set_ylabel('actual')
plt.suptitle('gradient boosting confusion matrices', fontsize=13)
plt.tight_layout()
plt.savefig('gb_confusion_matrices.png', dpi=150)
plt.show()
In [24]:
fi_gb = pd.DataFrame({
'feature': X_train_enc.columns,
'importance': best_gb.feature_importances_
}).sort_values('importance', ascending=False).head(15)
plt.figure(figsize=(10, 6))
plt.barh(fi_gb['feature'], fi_gb['importance'], color='seagreen')
plt.gca().invert_yaxis()
plt.xlabel('importance score')
plt.title('top 15 features — gradient boosting')
plt.tight_layout()
plt.savefig('gb_feature_importance.png', dpi=150)
plt.show()
print("\ntop 15 features:")
print(fi_gb.to_string(index=False))
top 15 features:
feature importance
src_bytes 0.338919
dst_host_serror_rate 0.147108
same_srv_rate 0.136413
dst_bytes 0.092748
count 0.067477
dst_host_diff_srv_rate 0.050019
dst_host_srv_diff_host_rate 0.034358
dst_host_same_src_port_rate 0.025258
diff_srv_rate 0.024927
protocol_type 0.023652
dst_host_srv_count 0.018813
wrong_fragment 0.008474
dst_host_rerror_rate 0.006929
num_compromised 0.006292
srv_count 0.003009
6. Model Comparison¶
In [25]:
comparison = pd.DataFrame({
'model': ['Random Forest', 'Gradient Boosting'],
'accuracy': [rf_acc, gb_acc],
'macro_f1': [rf_f1, gb_f1],
'best_params': [str(rf_grid.best_params_), str(gb_grid.best_params_)]
})
display(comparison.style.format({
'accuracy': '{:.4f}',
'macro_f1': '{:.4f}'
}))
| model | accuracy | macro_f1 | best_params | |
|---|---|---|---|---|
| 0 | Random Forest | 0.9300 | 0.7233 | {'max_depth': None, 'n_estimators': 100} |
| 1 | Gradient Boosting | 0.9306 | 0.7163 | {'learning_rate': 0.1, 'max_depth': 2, 'n_estimators': 150} |
In [26]:
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
colours = ['steelblue', 'seagreen']
models = comparison['model'].tolist()
for ax, metric, label, ylim in zip(
axes,
['accuracy', 'macro_f1'],
['accuracy', 'macro F1'],
[(0.88, 1.0), (0.60, 1.0)]
):
bars = ax.bar(models, comparison[metric], color=colours)
ax.set_ylim(ylim)
ax.set_ylabel(label)
ax.set_title(f'{label} comparison')
for bar, val in zip(bars, comparison[metric]):
ax.text(bar.get_x() + bar.get_width() / 2,
val + (ylim[1] - ylim[0]) * 0.01,
f'{val:.4f}', ha='center', fontsize=10)
plt.suptitle('model comparison: random forest vs gradient boosting', fontsize=13)
plt.tight_layout()
plt.savefig('model_comparison.png', dpi=150)
plt.show()
In [27]:
# extract per-class f1 from classification reports for direct comparison
from sklearn.metrics import classification_report as cr
rf_report = cr(y_test, y_pred_rf, output_dict=True)
gb_report = cr(y_test, y_pred_gb, output_dict=True)
classes = ['dos', 'normal', 'probe', 'r2l', 'u2r']
rf_f1_per_class = [rf_report[c]['f1-score'] for c in classes]
gb_f1_per_class = [gb_report[c]['f1-score'] for c in classes]
x = np.arange(len(classes))
width = 0.35
fig, ax = plt.subplots(figsize=(10, 5))
ax.bar(x - width/2, rf_f1_per_class, width, label='random forest', color='steelblue')
ax.bar(x + width/2, gb_f1_per_class, width, label='gradient boosting', color='seagreen')
ax.set_xticks(x)
ax.set_xticklabels(classes)
ax.set_ylabel('f1 score')
ax.set_ylim(0, 1.05)
ax.set_title('per-class f1 score: random forest vs gradient boosting')
ax.legend()
ax.axhline(y=0.5, color='red', linestyle='--', linewidth=0.8, alpha=0.5)
for i, (v1, v2) in enumerate(zip(rf_f1_per_class, gb_f1_per_class)):
ax.text(i - width/2, v1 + 0.01, f'{v1:.2f}', ha='center', fontsize=8)
ax.text(i + width/2, v2 + 0.01, f'{v2:.2f}', ha='center', fontsize=8)
plt.tight_layout()
plt.savefig('per_class_f1_comparison.png', dpi=150)
plt.show()
In [28]:
top_features = fi_rf['feature'].tolist()
fi_gb_aligned = (
pd.DataFrame({'feature': X_train_enc.columns,
'importance': best_gb.feature_importances_})
.set_index('feature')
.reindex(top_features)
.reset_index()
)
x = np.arange(len(top_features))
width = 0.35
fig, ax = plt.subplots(figsize=(13, 6))
ax.bar(x - width/2, fi_rf['importance'].values, width,
label='random forest', color='steelblue')
ax.bar(x + width/2, fi_gb_aligned['importance'].values, width,
label='gradient boosting', color='seagreen')
ax.set_xticks(x)
ax.set_xticklabels(top_features, rotation=45, ha='right', fontsize=9)
ax.set_ylabel('importance score')
ax.set_title('feature importance comparison — top 15 RF features')
ax.legend()
plt.tight_layout()
plt.savefig('feature_importance_comparison.png', dpi=150)
plt.show()
In [29]:
y_pred_rf_train = best_rf.predict(X_train_scaled)
y_pred_gb_train = best_gb.predict(X_train_scaled)
rf_train_f1 = f1_score(y_train, y_pred_rf_train, average='macro')
gb_train_f1 = f1_score(y_train, y_pred_gb_train, average='macro')
print("overfitting analysis — macro F1 (train vs test)")
print(f"{'model':<22} {'train f1':>10} {'test f1':>10} {'gap':>10}")
print("-" * 54)
print(f"{'random forest':<22} {rf_train_f1:>10.4f} {rf_f1:>10.4f} {rf_train_f1 - rf_f1:>+10.4f}")
print(f"{'gradient boosting':<22} {gb_train_f1:>10.4f} {gb_f1:>10.4f} {gb_train_f1 - gb_f1:>+10.4f}")
print()
print("interpretation:")
print(" gap near 0 = good generalisation")
print(" large positive = overfitting (model memorises training data)")
print(" negative = test performance exceeds training (unlikely, check data)")
overfitting analysis — macro F1 (train vs test) model train f1 test f1 gap ------------------------------------------------------ random forest 0.9980 0.7233 +0.2748 gradient boosting 0.9032 0.7163 +0.1869 interpretation: gap near 0 = good generalisation large positive = overfitting (model memorises training data) negative = test performance exceeds training (unlikely, check data)
7. Summary¶
The table below consolidates the key evaluation metrics for both models alongside their computational characteristics, providing a basis for the recommendation made in the accompanying report.
In [30]:
summary = pd.DataFrame({
'metric': ['accuracy', 'macro f1', 'dos f1', 'probe f1',
'r2l f1', 'u2r f1', 'train f1 (overfit check)',
'training time (relative)'],
'random forest': [f'{rf_acc:.4f}',
f'{rf_f1:.4f}',
f'{rf_report["dos"]["f1-score"]:.4f}',
f'{rf_report["probe"]["f1-score"]:.4f}',
f'{rf_report["r2l"]["f1-score"]:.4f}',
f'{rf_report["u2r"]["f1-score"]:.4f}',
f'{rf_train_f1:.4f}',
'fast (parallel)'],
'gradient boosting': [f'{gb_acc:.4f}',
f'{gb_f1:.4f}',
f'{gb_report["dos"]["f1-score"]:.4f}',
f'{gb_report["probe"]["f1-score"]:.4f}',
f'{gb_report["r2l"]["f1-score"]:.4f}',
f'{gb_report["u2r"]["f1-score"]:.4f}',
f'{gb_train_f1:.4f}',
'slow (sequential)'],
})
display(summary.set_index('metric'))
| random forest | gradient boosting | |
|---|---|---|
| metric | ||
| accuracy | 0.9300 | 0.9306 |
| macro f1 | 0.7233 | 0.7163 |
| dos f1 | 0.9614 | 0.9649 |
| probe f1 | 0.9206 | 0.9032 |
| r2l f1 | 0.3173 | 0.3218 |
| u2r f1 | 0.4800 | 0.4528 |
| train f1 (overfit check) | 0.9980 | 0.9032 |
| training time (relative) | fast (parallel) | slow (sequential) |
In [ ]: