System Architecture
This pipeline demonstrates a production-ready streaming system that ingests clickstream events from Kafka, processes and stores them in Delta Lake (Bronze → Silver → Gold), and performs anomaly detection using MLflow.
🧱 End-to-End Data Flow
This architecture is built on Databricks and leverages the medallion (Bronze/Silver/Gold) architecture pattern to progressively refine streaming event data.
Diagram: Real-time events flow from Kafka → Delta Bronze → Silver → MLflow → Gold Delta Table
Orchestrated with Databricks Workflows
The pipeline is orchestrated with Databricks Workflows, which includes separate tasks for ingesting Kafka events, transforming data, running batch inference, and writing scored outputs to Gold.
Workflow DAG: Kafka ingestion → Transformation → ML Inference → Gold Output
MLflow Integration
MLflow powers the model lifecycle management and batch inference logic. This includes:
- Model Registry: A versioned Isolation Forest model used for anomaly detection
- Batch Scoring: Spark UDF applies the model on Silver table events
- MLflow Logging: Logs
avg_anomaly_score,run_id, andsample_predictions.json - Auditing: Delta table lineage is tracked with
DESCRIBE HISTORY
System Highlights
- Real-time ingestion: Using Kafka + Structured Streaming into Bronze
- Feature Engineering: VectorAssembler on event value and timestamp
- Delta Lake: Partitioned storage with audit-ready schema evolution
- ML Inference: Isolation Forest via MLflow PyFunc Spark UDF
- Scalability: Designed to support streaming and batch scoring at scale