ARMS Kafka Workload Classifier

A Logistic Regression model that classifies an Apache Kafka cluster's current workload as real-time / event-driven or batch / data-intensive from four live broker and host metrics. It is the decision component of ARMS (Adaptive Recovery Management System). ARMS uses the prediction to choose a Kafka recovery strategy when a broker fails, instead of applying one static configuration to every workload.

  • Code, Docker stack and experiments: github.com/NwaoghorPraise2/ARMS-project
  • Author: Nwaoghor Praise Chukunweiken
  • Context: MSc Software Engineering thesis, University of Hertfordshire (April 2025)
  • Model type: scikit-learn LogisticRegression + StandardScaler (two joblib files)
  • License: Apache 2.0

Files

File What it is
kafka_logistic_regression_model.pkl Trained LogisticRegression (random_state=42, max_iter=1000)
kafka_feature_scaler.pkl Fitted StandardScaler. Scale your inputs with it before predicting.
inference.py Download, classify and pick a strategy in one script
requirements.txt Python dependencies for inference.py
confusion_matrix.png, feature_importance.png, cross_validation.png Evaluation figures

Inputs

The model expects exactly these four features, in this order. In ARMS each one comes from a Prometheus query (Kafka JMX exporter + node-exporter):

# Feature Prometheus query used in ARMS
1 request_rate_variance stddev_over_time(sum(rate(kafka_server_brokertopicmetrics_totalproducerequests_total[1m]))[15m:1m])
2 node_disk_write_bytes_total sum(rate(node_disk_written_bytes_total[1m]))
3 message_rate_variance stddev(rate(kafka_server_brokertopicmetrics_messagesin_total[1m]))
4 node_memory_MemAvailable_bytes avg(node_memory_MemAvailable_bytes)

Outputs

Class Label Recovery strategy ARMS selects (confidence โ‰ฅ 90%)
0 REAL_TIME_EVENT_DRIVEN QUICK_REBALANCE: fast leader rebalancing, prioritises recovery speed
1 BATCH_DATA_INTENSIVE RESOURCE_OPTIMIZED: slower rebalancing, prioritises resource usage
either, confidence < 90% ambiguous CONTROLLED_GRADUAL: cautious default

How to use

Quick start

pip install "scikit-learn>=1.6,<1.7" joblib pandas huggingface_hub
import joblib
import pandas as pd
from huggingface_hub import hf_hub_download

repo = "nwaoghorpraise/arms-kafka-workload-classifier"
model = joblib.load(hf_hub_download(repo, "kafka_logistic_regression_model.pkl"))
scaler = joblib.load(hf_hub_download(repo, "kafka_feature_scaler.pkl"))

features = [
    "request_rate_variance",
    "node_disk_write_bytes_total",
    "message_rate_variance",
    "node_memory_MemAvailable_bytes",
]

snapshot = pd.DataFrame([{
    "request_rate_variance": 96.19,
    "node_disk_write_bytes_total": 5.50e7,
    "message_rate_variance": 235.35,
    "node_memory_MemAvailable_bytes": 5.176e9,
}], columns=features)

proba = model.predict_proba(scaler.transform(snapshot))[0]
label = int(proba.argmax())
print({0: "REAL_TIME_EVENT_DRIVEN", 1: "BATCH_DATA_INTENSIVE"}[label], f"{proba[label]:.2%}")
# REAL_TIME_EVENT_DRIVEN 99.99%

With strategy selection

inference.py wraps the same steps and applies the ARMS 90%-confidence rule:

python inference.py
# {'workload': 'REAL_TIME_EVENT_DRIVEN', 'confidence': 0.9999, 'strategy': 'QUICK_REBALANCE'}
# {'workload': 'BATCH_DATA_INTENSIVE', 'confidence': 0.989, 'strategy': 'RESOURCE_OPTIMIZED'}

Live, against a real Kafka cluster

To classify a running cluster, run the full ARMS stack from the GitHub repo (3-broker Kafka, Prometheus, Grafana, classifier, controller). The classifier polls Prometheus every 30 seconds and serves its result at http://localhost:5001/api/prediction and /api/strategy:

git clone https://github.com/NwaoghorPraise2/ARMS-project
cd ARMS-project
./scripts/setup.sh && ./scripts/start.sh
./scripts/run_experiment.sh real_time   # or: batch

Training

  • Data: 3,000 labelled metric snapshots (1,500 per class) collected from a 3-broker Kafka cluster running synthetic real-time and batch AI workloads (datasets/balanced_kafka_dataset_3000.csv in the GitHub repo).
  • Feature selection: 15 candidate metrics ranked with mutual information, Spearman correlation and Random Forest importance (metric_analysis.py). The four features above were kept.
  • Split: stratified 80/20 train/test, random_state=42. Features standardised with StandardScaler, fitted on the training set.
  • Reproduce: python train_model.py in the GitHub repo.

Evaluation

On the 600-sample hold-out set:

Precision Recall F1 Support
REAL_TIME_EVENT_DRIVEN (0) 0.9727 0.9500 0.9612 300
BATCH_DATA_INTENSIVE (1) 0.9511 0.9733 0.9621 300
Accuracy 0.9617 600
  • 5-fold cross-validation accuracy: 0.9677
  • Confusion matrix: 285 / 15 / 8 / 292 (rows = true class)
  • Inference: well under 0.1 ms per sample on CPU

Effect in the full ARMS system (thesis results): a mean reduction of 6.14 s in recovery time for real-time workloads (p < 0.001), no significant change for batch workloads (p = 0.9986), about 15% higher memory use, and about 3% lower CPU use for real-time workloads.

Confusion matrix Feature importance

Limitations

  • Trained on one cluster. The data comes from a single Docker-based 3-broker setup with synthetic workloads. The absolute feature values, such as available memory around 5.1 GB and disk write rates, reflect that environment. Re-train on your own cluster's metrics before relying on it in production.
  • Two classes only. Mixed or unusual workloads are forced into one of the two classes. The 90% confidence fallback to CONTROLLED_GRADUAL exists to catch these cases.
  • Pickle format. The files are joblib/pickle and were saved with scikit-learn 1.6.1. Load them with a matching 1.6.x version, and only load pickles from sources you trust.

Citation

@mastersthesis{chukunweiken2025arms,
  author = {Nwaoghor Praise Chukunweiken},
  title  = {Optimising Recovery in AI Workload Pipeline Message Brokering Systems Using Machine Learning: Adaptive Strategies for Time and Resource Efficiency},
  school = {University of Hertfordshire},
  year   = {2025},
  month  = {April},
  type   = {MSc Software Engineering thesis}
}
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Evaluation results

  • Test accuracy (20% hold-out) on ARMS balanced Kafka workload dataset (3,000 samples)
    self-reported
    0.962
  • Macro F1 on ARMS balanced Kafka workload dataset (3,000 samples)
    self-reported
    0.962
  • 5-fold cross-validation accuracy on ARMS balanced Kafka workload dataset (3,000 samples)
    self-reported
    0.968