Introduction
This project is an Intrusion Detection System (IDS) that allows users to upload network data files for Machine Learning (ML) inference. The system pre-processes the uploaded file, runs it through an ML model trained on the NSL-KDD dataset, and returns the classification results to the frontend. Processed results are stored in S3, and job status is tracked in DynamoDB.
The architecture is built using a Microservices architecture hosted on AWS Fargate (ECS) with an internal and external load balancer setup, ensuring secure and scalable operations.
Objectives
The primary goals of the IDS project are:
Data Pre-processing: Transform raw network traffic data (based on the NSL-KDD dataset) into a structured format for machine learning.
Model Training: Compare multiple machine learning models to identify the most accurate and efficient one for intrusion detection.
User Interaction: Build a user-friendly web interface using Flask for users to upload network data for classification.
File Classification: Build the application that will classify the uploaded file based on the machine learning model.
Cloud Deployment: Deploy the entire system using AWS ECS Fargate, Terraform, and a robust CI/CD pipeline for automation.
Scalability: Ensure the system can handle increased traffic and adapt to evolving threats.
CI/CD Pipeline: Create a CI/CD pipeline that’ll ensure seamless deployment of code changes.
🛠️ Tech Stack
Machine Learning: Pre-processing with Pandas and NumpPy, and training models with SciKit-learn.
Frontend: NextJS frontend
Backend: Flask-based web application for user interaction.
Infrastructure as Code (IaC): Terraform for provisioning AWS resources.
CI/CD: Automated deployment pipeline (To Be Determined).
Cloud Deployment: Containerized application.
Visualization: Generate real-time classification insights with Matplotlib.
🚦 Key Features
Efficient Data Processing: Leverages advanced preprocessing techniques to prepare data for model training.
Multi-Model Comparison: Tested models such as Random Forest, Decision Tree, Naive Bayes, and Logistic Regression to ensure the best performance.
Interactive User Interface: Upload network traffic data and receive immediate threat classification.
Cloud-Native Architecture: Deployed using AWS services for high availability and scalability.
🔍 Project Workflow
Data Preprocessing: Cleaned and transformed the NSL-KDD dataset into machine-readable formats.
Model Training: Trained multiple models to classify threats, comparing metrics like accuracy, precision, recall, and F1 score to select the best model.
Model Deployment: Packaged the trained model as a Docker container and pushed it to Amazon ECR.
Web App Development: Created a Flask-based app for user interactions.
Cloud Deployment: Deployed the app on AWS ECS Fargate, ensuring security and scalability.
🏆 What’s Next?
This is just the beginning! In future posts, I’ll dive deeper into:
Enhancing the dataset by incorporating real-world scenarios.
Improving the Flask app with user authentication and a database for result tracking.
Scaling the system with Kubernetes for even greater flexibility.
Exploring real-time anomaly detection using advanced ML techniques.
🔗 Stay Tuned!
This post is part of my Projects series. For all updates and technical details on this Intrusion Detection System, check out my Intrusion Detection System series, where I'll break down each component and share insights along the way.
Thank you for following my journey! Feel free to share your thoughts or ask questions in the comments. 😊
