Real-time object detection and classification system that identifies personal belongings on a conveyor belt and communicates the object type to the IoT system for automated sorting into the correct bin.
- Project Overview
- Problem Statement
- Goals & Objectives
- System Architecture
- Classes
- Project Phases
- YOLO Model Comparison
- Iterative Improvement Log
- Results
- Tech Stack
- IoT Integration
- What's Next
- Team
SpectraSense is an AI-powered smart sorting system designed to classify personal belongings — the kind of items people typically carry with them — and route them automatically to the correct bin using a conveyor belt controlled by an IoT system.
The concept is inspired by recycling systems, but instead of sorting by material alone, SpectraSense goes one level deeper: it identifies what the object actually is, determines its identity and metal content, then communicates that classification to the hardware for physical sorting.
A camera positioned above the conveyor belt captures each object as it passes. The AI model identifies it in real time and sends the predicted label to the IoT system via ESP. Based on the label, the conveyor and servo motors activate to deliver the object to one of four designated bins.
| Bin | Category | Objects |
|---|---|---|
| 1 | Metal Electronics | AirPods, AirPods case, charger, headphones, mobile phone, power bank |
| 2 | Metal Accessories | Wrist watch, necklace, ring |
| 3 | Metal Tools | Key, coin, bank card |
| 4 | Not Metal | Wallet, ID card, pen, pencil, bottle |
Current automated sorting systems often stop at material recognition — distinguishing metal from plastic, for example. However, knowing the material alone is not enough for advanced logistics or security applications. A coin and a key are both metal, yet they require completely different handling. A bank card and an ID card look nearly identical yet belong to different categories.
There is a need for a multi-layered identification system that recognizes both the substance and the specific identity of the object — enabling smarter, more precise automated sorting that goes beyond what material sensors alone can achieve.
Develop a system that accurately identifies and categorizes each object — not just its material, but its specific class — enabling precise bin assignment for every item.
Train the model using objects captured from diverse angles, orientations, and realistic perspectives to mimic actual operating conditions on a conveyor belt environment.
Calibrate the model to account for specific camera positioning and the expected nature of items, ensuring seamless transition from training to real-time deployment.
- Data Refinement: Curate high-quality samples by removing noise and redundant data
- Label Optimization: Merge similar classes to eliminate model confusion
- Enhanced Robustness: Diversify training data to cover complex real-world variations
┌──────────────────────────────────────────────────────────────────────────┐
│ FULL PIPELINE │
│ │
│ Object placed Camera captures YOLO model detects │
│ on conveyor ──► object frame ──► & classifies object ──► │
│ label + confidence │
│ │
│ ──► Label sent IoT receives Conveyor + servos │
│ to ESP ──► label via ESP ──► deliver object to ──► │
│ correct bin │
└──────────────────────────────────────────────────────────────────────────┘
Camera Frame (live feed)
│
▼
┌───────────────┐
│ YOLO v26n │ ◄── Detects object & predicts class in real time
│ Detection │ Trained on multi-source labeled dataset
└───────┬───────┘
│
│ Bounding box + Class label + Confidence score
▼
┌───────────────┐
│ OpenCV Live │ ◄── Draws annotations on video stream
│ Display │ Bounding box · Label · Confidence %
└───────┬───────┘
│
▼
Label sent to ESP
│
▼
IoT System → Conveyor + Servos → Correct Bin
All classes represent objects people commonly carry daily, grouped into the four sorting categories:
AirPods · AirPods Case · Charger · Headphones · Mobile Phone · Power Bank
Wrist Watch · Necklace · Ring
Key · Coin · Bank Card
Wallet · ID Card · Pen · Pencil · Bottle
During the research phase, we evaluated the following candidates before finalizing the above:
AirPods · Headphones · AirPods Case · Mobile Phone · Smart Watch · Bank Card · ID Card · Bottle · Glasses / Sunglasses · Wallet · Pen · Pencil · Eraser · Necklace · Bracelet · Ring · Charger · Power Bank · Coin · Key · Paper · Cups
Before collecting a single image, the most critical decision was what to detect. Not all object classes are created equal — some are too rare, some are too ambiguous, and some would not realistically appear in a conveyor belt scenario.
Our selection criteria were clear: we focused on objects that people commonly carry with them on a daily basis — items that would realistically appear in a camera frame in public or checkpoint environments. This included things like mobile phones, keys, wallets, bottles, and similar everyday items.
With our classes defined, the next step was gathering raw image data at scale. A YOLO model learns by seeing at least hundreds of examples of each object, from different angles, lighting conditions, backgrounds, and distances. A small or narrow dataset produces a fragile model — one that works in the lab but fails in the real world.
We pulled data from multiple platforms:
| Source | Description |
|---|---|
| Roboflow | Pre-labeled computer vision datasets with existing bounding box annotations |
| Kaggle | Community-shared image collections |
| Other repositories | Various open-source datasets for specific classes |
Datasets collected per class:
| Class Group | Source |
|---|---|
| Phone · Smart Watch · Earphone | Kaggle / Roboflow |
| Headphone · Power Bank | Kaggle / Roboflow |
| Phone · Bank Card | Roboflow |
| ID / Bank Card (×4 datasets) | Roboflow |
| Jewelry | Roboflow |
| Sunglasses & Glasses | Kaggle / Roboflow |
| Cell Phones | Kaggle |
| Paper · Bottle · Cup | Roboflow |
| Wallet (×2 datasets) | Roboflow |
| Coin · Key · Pen | Kaggle / Roboflow |
| Charger (×2 datasets) | Roboflow |
| Various multi-class datasets | Roboflow / Kaggle |
Using multiple sources was intentional — it ensures variety in image quality, resolution, scene context, and geographic diversity.
Raw data is never ready for training. After collection, we went through a rigorous filtering process — removing blurry images, duplicates, mislabeled examples, and images where the object was too small or partially visible to be useful. Quality always outweighs quantity in computer vision.
All accepted images were uploaded into a shared Roboflow workspace, where we managed labeling, versioning, and dataset splits. For images from Kaggle or other sources without bounding box annotations, we drew every bounding box manually — image by image — ensuring each label was tight, accurate, and consistent across the team.
⚠️ Manual labeling is the most time-consuming step in this project — but it is the direct foundation of model accuracy. One consistently mislabeled class can corrupt an entire detection category.
With a clean, labeled dataset in place, we moved into model training. We chose the YOLO (You Only Look Once) family for its exceptional balance between detection speed and accuracy — making it ideal for real-time applications.
Models evaluated:
| Model | Outcome |
|---|---|
| YOLO 11n | Tested — problems with Bracelet, Key, Necklace, Pen, Pencil, Sunglasses |
| YOLO 26n | ✅ Selected as final model |
| YOLO 26s | Tested — compared against 26n |
| YOLO 26m | Tested — compared against 26n |
Final model: YOLO 26n
The model was trained with parameters configured for our specific use case — including image size, batch size, number of epochs, and confidence thresholds. Training was monitored continuously to catch signs of overfitting or underfitting early.
Once training completed, we moved into structured evaluation. Testing is not simply running the model on random images — it requires using a dedicated test set: images the model has never seen during training, carefully separated before the training process began.
We evaluated the model's performance on a per-label basis, meaning each object class was assessed individually rather than just looking at an overall average. This approach reveals:
- Which specific classes the model handles confidently ✅
- Which ones require targeted improvement
⚠️ - Whether issues are caused by insufficient examples, visual similarity, or inconsistent labeling
Achieving strong accuracy on the first training run is rare. Model improvement is an iterative process — a cycle of training, evaluating, identifying weak points, making targeted changes, and re-training.
Our benchmark: most labels must reach ≥ 89% accuracy
When a class fell below the threshold, we had two distinct strategies:
| Strategy | Action |
|---|---|
| Model-side | Switch to a different YOLO version or adjust hyperparameters |
| Data-side | Add diverse images, remove ambiguous examples, combine similar sub-classes, augment existing data |
💡 Most accuracy gains came from better data, not a better model. When a class underperformed, the data was always the first place we looked.
Before any final deployment, we conducted a comprehensive review of the entire dataset. After many rounds of additions, deletions, and modifications across multiple iteration cycles, it is critical to ensure the dataset is internally consistent:
- ✅ No outdated labels remaining
- ✅ No conflicting class definitions
- ✅ Train / validation / test splits correctly balanced
- ✅ Labeling conventions applied uniformly across all classes
- ✅ Final dataset version locked in Roboflow
The final and most revealing test of any computer vision model is not on a static image dataset — it is in the real world, in real time.
We integrated our trained YOLO model with OpenCV — a powerful open-source computer vision library that allows us to access a live camera feed, process each frame, and display detection results in real time.
Every frame captured by the camera is passed through the model, which draws bounding boxes, class labels, and confidence scores directly on the video stream. This live test reveals how the model handles:
- Real lighting conditions and shadows
- Motion blur from a moving conveyor belt
- Partial occlusions of objects
- The natural unpredictability of a live environment
💡 If a model performs well on static test images but struggles on live camera, the cause is usually motion blur or insufficient variety in training data.
| Model | Status | Notes |
|---|---|---|
| YOLO 11n | ❌ Not selected | Problems with: Bracelet, Key, Necklace, Pen, Pencil, Sunglasses |
| YOLO 26n | ✅ Final model | Best overall performance across all classes |
| YOLO 26s | Tested | Compared against 26n |
| YOLO 26m | Tested | Compared against 26n |
Each iteration addressed specific underperforming classes. Below is the full log of changes made across all training runs:
| Run | Model | Changes Made |
|---|---|---|
| 1 | YOLO 11n | Initial training — problems: Bracelet, Key, Necklace, Pen, Pencil, Sunglasses |
| 2 | YOLO 26n | Removed Bracelet · Edited ID Card class · Added new data for Key · Added Eraser |
| 3 | YOLO 26n | Combined Jewelry classes · Added new data for Key & Sunglasses |
| 4 | YOLO 26n | Added new data for Key & Sunglasses · Combined Pen & Pencil into one class |
| 5 | YOLO 26n | Added Calculator & Notebook classes |
| 6 | YOLO 26n | Improved Pens/Pencils class data |
| 7 | YOLO 26n | Problems remaining: Eraser, Key, Sunglasses |
| 8 | YOLO 26n | Removed Eraser · Problems remaining: Key, Sunglasses |
| Metric | Value |
|---|---|
| Final model | YOLO 26n |
| Per-class accuracy target | ≥ 89% for most labels |
| Inference type | Real-time, live camera feed via OpenCV |
| Integration | OpenCV display + ESP communication to IoT system |
| Tool / Library | Purpose |
|---|---|
| YOLO 26n | Real-time object detection and classification |
| OpenCV | Live camera feed processing and bounding box annotation |
| Roboflow | Dataset management, labeling, versioning, augmentation, splits |
| Python | Training pipeline, inference scripts, ESP communication |
| Kaggle | Data source for image collection |
📌 This section to be completed by the IoT team
The AI model communicates predicted labels to the IoT system via ESP (ESP8266 / ESP32).
Communication flow:
YOLO Model predicts class
│
│ Label + Confidence Score
▼
ESP Module
│
├──► Conveyor belt motor (move to correct position)
└──► Servo motors ──► Route object into correct bin
Sorting logic:
| Predicted Class | Bin |
|---|---|
| AirPods, AirPods Case, Charger, Headphones, Mobile Phone, Power Bank | Bin 1 — Metal Electronics |
| Wrist Watch, Necklace, Ring | Bin 2 — Metal Accessories |
| Key, Coin, Bank Card | Bin 3 — Metal Tools |
| Wallet, ID Card, Pen, Pencil, Bottle | Bin 4 — Not Metal |
[IoT team: please add hardware specifications, component list, wiring diagram, firmware details, and ESP communication protocol here]
Our project does not end here — it marks the beginning of a larger vision. The current system successfully detects everyday objects in real time, but there is significant room to grow in both capability and impact.
Expand detection coverage by introducing additional classes beyond the current set — going through the same rigorous collection, filtering, and labeling pipeline already established.
Develop a graphical interface that allows real people to try the system live — removing the lab environment and exposing the model to real-world diversity of users, objects, and conditions.
Integrate a built-in survey form after each session so users can flag misclassifications and suggest improvements — turning every interaction into a data point for the next version.
Compress and optimize the model using quantization and pruning techniques for deployment on Raspberry Pi, mobile phones, or embedded security modules — enabling portable and low-power applications.
Move beyond category recognition toward full object identity — not just "this is metal" but "this is specifically a key" — enabling smarter handling decisions at every level of the sorting pipeline.
| Name | Role | GitHub | |
|---|---|---|---|
| [Mohammed Abdelnaser] | Team Lead | GitHub | [LinkedIn] |
| Mohammed Gamal | Member | GitHub | [LinkedIn] |
| [Hager Ali] | Member | GitHub |
| Name | Role | GitHub | |
|---|---|---|---|
| [Malak] | Team Lead | GitHub | |
| [Manar Ashraf] | Member | GitHub | |
| Jana Kassem | Member | GitHub | |
| Habiba Elsayed | Member | GitHub |
[Add supervisor names and titles here]
[Add names of anyone who supported or mentored the project here]
| Event | Details |
|---|---|
| 🤖 RoboTech Fair | [26 / 4 , 27 / 4] |
| 🏆 AZEX Competition | [4 / 5] |
SpectraSense — Smart Object Sorting System
AI Component · YOLO 26n · OpenCV · Roboflow · Python