IDS Defense – SOC-Grade ML Intrusion Detection
GitHubAddressed lack of real-time visibility into network attacks faced by SOC teams handling high-volume logs.
Overview
IDS Defense is an ML-powered log analysis pipeline that ingests raw network traffic, extracts features, and scores them using a trained XGBoost model to detect anomalies in real-time.
The Problem
Network security teams lack real-time visibility into complex attack vectors, overwhelmed by raw logs without actionable insights or ML context.
System Architecture
Python/FastAPI ingestion API that pushes logs to a message queue. Worker nodes pull logs, perform feature engineering, and run XGBoost inference, pushing alerts to a Next.js frontend via WebSockets.
Engineering Trade-offs
Used XGBoost instead of Deep Learning to drastically reduce inference time and compute costs, accepting a slightly higher false-positive rate on novel attacks.
Lessons Learned & Future
Production Learnings
Feature extraction (like parsing User-Agents and IP geolocation) is often slower than the ML inference itself. Moving extraction to a compiled language (Rust) would be a logical next step.
Future Improvements
Implementing a continuous feedback loop where SOC analysts can flag false positives to retrain the model dynamically.
Database Layer
Elasticsearch for high-speed log querying and aggregation. Redis for the message queue.
API Design
Bulk ingestion REST API for log shippers (like Filebeat). WebSocket API for streaming alerts to the SOC dashboard.
Security Decisions
1. Implemented mutual TLS (mTLS) for all log shipper connections to prevent log spoofing. 2. The SOC dashboard requires MFA for access.
Scale & Metrics
Pipeline successfully ingests and scores 10,000 logs per second on consumer hardware.