Portfolio

Warehouse Safety Monitoring System

The Warehouse Safety Monitoring System is a computer vision-driven analytics platform that reviews CCTV footage of warehouse aisles and produces structured, human readable safety telemetry in near real time. Rather than relying on a single detection model, the system uses a two stage hybrid pipeline: a fast computer-vision detector localizes people and vehicles in every sampled frame, and a large language model (LLM) reasoning layer interprets those detections in context generating natural language scene descriptions, activity labels, and PPE (Personal Protective Equipment) compliance judgments such as hard hat and hi-vis vest status.

The result is a dashboard that lets a safety officer scrub through footage, trigger an on-demand AI analysis of any frame, and immediately see a composite safety score, object census and per worker compliance breakdown without manually reviewing hours of raw video.

The Challenge

1 Compute vs. depth trade-off running a full vision-language reasoning pass on every frame of a continuous feed is too slow and expensive for real-time use.

2 Semantic ambiguity in PPE compliance a bounding box alone can't reliably determine "wearing a hard hat" under occlusion, motion blur, or off-angle camera views; this requires contextual judgment, not just classification.

3 Resolution inconsistency footage arrives from cameras with different resolutions and aspect ratios, so raw pixel-coordinate bounding boxes don't transfer cleanly across sources.

4 Output reliability LLM generated descriptions are inherently less structured than classifier outputs; the system needs a guaranteed, parseable contract for the frontend to render.

5 False positives in a high-stakes domain a safety system that cries wolf gets ignored; confidence needs to be visible and tunable, not hidden.

6 Actionability, not just detection raw detections are only useful if they roll up into something a safety officer can act on in seconds, not minutes.

Our Solution

The core design decision was to not ask one model to do everything. Instead, detection and reasoning were split into two specialized stages that play to each approach's strengths:

Stage 1: CV Detection (fast, always-on): A dedicated object detector [e.g. YOLOv8 / Detectron2] runs on sampled frames to localize people and vehicles quickly and cheaply, returning class labels, confidence scores, and bounding boxes. This is what powers continuous timeline scrubbing without incurring reasoning-model latency on every frame.

Stage 2: LLM Reasoning (on-demand, contextual): When a user (or an automated trigger) requests deeper analysis of a specific frame, the detected regions are passed to a vision-capable LLM reasoning layer. This stage produces the parts that plain object detection can't: a natural-language scene summary, per-object activity labels, and semantic PPE compliance calls (hard hat / hi-vis pass-fail) that account for context a classifier would miss.

This mirrors the "Analyze This Frame (AI)" affordance in the UI analysis is event-triggered rather than continuous, which was a deliberate cost/latency control rather than a limitation

Technical Architecture Highlights

┌───────────────────┐ ┌───────────────────────┐ ┌──────────────────────────┐

│ Video Ingestion │ │ Stage 1: CV Detection │ │ Stage 2: LLM Reasoning │

│ & Frame Sampling │────▶│ [YOLO / Detectron2] │────▶│ [Vision-LLM API] │

└───────────────────┘ └───────────────────────┘ └──────────────────────────┘

│ │

▼ ▼

class + confidence scene description,

+ raw bounding box activity label,

PPE compliance flags

│ │

└──────────────┬───────────────┘

▼

┌───────────────────────────────┐

│ Fusion & Normalization Layer │

│ • 0–1000 coordinate grid │

│ • Schema validation │

│ • Composite safety scoring │

└───────────────────────────────┘

│

▼

┌───────────────────────────────┐

│ Telemetry API / Dashboard │

│ • Video player + timeline │

│ • Bounding-box overlays │

│ • Live telemetry panel │

└───────────────────────────────┘

This page failed to load.

Refresh the page or re-upload the theme zip. If this persists, purge LiteSpeed/Cloudflare cache for HTML and theme assets.

Return home