Computer Vision in Production: From Tracking to VLMs to Edge Deployment

Computer vision has had a quietly transformative few years. The field that once meant "train a ResNet on ImageNet and call it a day" now spans real-time multi-object tracking at 60 fps, RAFT-based optical flow that beats classical methods at a fraction of the latency, Vision Language Models that answer open-ended questions about what a camera sees, and inference on purpose-built silicon that fits inside a dashcam housing. What hasn't changed is the gap between a working demo and a production system. This post is a practitioner's guide to crossing it — covering the tracking algorithms actually deployed at scale, the state of optical flow in 2026, how VLMs are reshaping CV pipelines, sensor fusion tradeoffs, and what it really takes to ship CV on constrained edge hardware.

1. Multi-Object Tracking: From SORT to ByteTrack

Multi-object tracking (MOT) is the problem of assigning consistent identities to detected objects across video frames — keeping track of person #7 as they move through a crowded retail aisle, or vehicle #23 as it merges onto a highway. It sounds simple. In practice, it is one of the harder CV engineering problems to get right at production quality, because it requires real-time performance, robustness to occlusion, and graceful handling of the "ID switch" failure mode where the tracker loses an identity and incorrectly reassigns it.

The tracking-by-detection paradigm

The dominant production approach remains tracking-by-detection: a detector (YOLO, RT-DETR, or similar) runs on each frame and produces a set of bounding boxes, which a separate tracker then links across frames using a combination of motion prediction and appearance features. This decoupled architecture lets you upgrade either component independently and is why most production MOT pipelines are built this way rather than on end-to-end learned approaches.

SORT (Simple Online and Realtime Tracking) established the baseline in 2016: Kalman filter for motion prediction, Hungarian algorithm for assignment. It runs at hundreds of frames per second but loses tracks frequently during occlusion. DeepSORT added a deep appearance feature extractor (a lightweight ReID network), dramatically reducing ID switches at the cost of higher compute. Both remain widely deployed where hardware is not severely constrained.

The current production standard for most new deployments is ByteTrack, which introduced a key insight: rather than discarding low-confidence detections (as SORT and DeepSORT do), ByteTrack uses them as a second association stage to recover occluded tracks. Low-confidence boxes that overlap strongly with existing predicted track positions are used to extend tracks through occlusion, before being confirmed by high-confidence detections when the object re-emerges. The result is substantially fewer ID switches with minimal additional computation — ByteTrack runs comfortably at real-time on a mid-range GPU and has become the default choice for retail analytics, surveillance, and sports tracking applications.

End-to-end trackers: the emerging challenger

End-to-end learned trackers — where detection and association are jointly optimized — have made serious progress. MOTR and its successors frame tracking as a set prediction problem, eliminating the hand-engineered association logic entirely. The appeal is clear: a single model that is aware of temporal context during training can learn to handle occlusion and reidentification in ways that a post-hoc association algorithm cannot. The barrier has been inference speed: end-to-end trackers based on transformer architectures are significantly heavier than ByteTrack + detector, and the gap matters when you are processing 8 camera streams on a single embedded board. As dedicated vision accelerators become more capable, this tradeoff will shift — but for most production deployments today, ByteTrack remains the pragmatic choice.

2. Optical Flow: Dense, Sparse, and RAFT

Optical flow — estimating per-pixel motion between frames — is one of those CV primitives that quietly underpins dozens of production applications. Video stabilization, action recognition, autonomous driving scene understanding, industrial inspection, surgical robotics: all of them use flow estimates, often implicitly.

Sparse vs dense flow

Sparse optical flow (Lucas-Kanade, tracked via the Kanade-Lucas-Tomasi feature tracker) computes motion for a set of keypoints rather than every pixel. It is fast, predictable, and still the right choice for camera pose estimation, SLAM front-ends, and video stabilization pipelines where dense coverage is unnecessary and compute is limited. OpenCV's calcOpticalFlowPyrLK remains production-standard for sparse flow.

Dense optical flow computes a motion vector for every pixel. Classical approaches (Farneback, TV-L1) produce smooth estimates but struggle with large displacements and fast motion. The deep learning era changed this decisively: FlowNet and PWC-Net showed that learned flow could outperform classical methods on standard benchmarks. But the real step-change was RAFT (Recurrent All-Pairs Field Transforms), introduced by Teed and Deng in 2020 and now refined into several production-oriented variants.

RAFT and its descendants in production

RAFT works by constructing an all-pairs correlation volume — comparing every patch in one frame against every patch in the other — and then iteratively refining a flow estimate using a GRU-based recurrent unit. This architecture delivers state-of-the-art accuracy on large-motion and fine-structure cases that defeat earlier networks. The tradeoff is memory: the all-pairs correlation volume is O(H×W×H×W), which is prohibitive at high resolution.

Production deployments that need RAFT-quality flow but can't afford its full memory footprint typically use one of two approaches: RAFT-Small (a distilled variant with roughly 5× fewer parameters, adequate for many video understanding tasks) or resolution-adaptive inference (full RAFT at downsampled resolution, with dense upsampling for the final map). For autonomous driving specifically, FlowFormer and scene-flow variants (which estimate 3D motion rather than 2D image-plane flow) are becoming the standard, as they integrate naturally with lidar and depth data.

Key production applications in 2026:

• Video stabilization: Sparse KLT flow for camera motion estimation + warp; deployed in every smartphone camera stack and most broadcast production systems
• Action recognition: Two-stream architectures (RGB + optical flow) still outperform pure RGB models on fine-grained actions — surgical gesture recognition, sports analytics, industrial assembly verification
• Autonomous driving: Dense flow used for ego-motion compensation, moving object segmentation, and feeding temporal context into downstream perception heads
• Industrial inspection: Flow-based anomaly detection catches defects that appear only in motion (vibration patterns, conveyor irregularities, subtle deformations under load)

3. Vision Language Models: How VLMs Are Reshaping CV Pipelines

The arrival of production-capable VLMs is the biggest structural shift in computer vision since the deep learning era began. Models like GPT-4V, LLaVA, Florence-2, and Qwen-VL can answer open-ended natural language questions about an image, generate descriptions, perform visual grounding (localizing objects described in text), and execute zero-shot classification without any task-specific training data. For production CV teams, this creates both opportunities and new engineering challenges.

Where VLMs actually belong in production pipelines

VLMs are not, in 2026, a replacement for task-specific CV models in latency-critical pipelines. A VLM adding textual reasoning to a video stream at 30 fps is not feasible with current inference economics — a GPT-4V API call takes hundreds of milliseconds and costs orders of magnitude more per frame than running YOLO locally. What VLMs have changed is the class of tasks that require custom ML development at all:

• Zero-shot quality inspection: "Does this product have a visible surface defect?" — asked over representative frames, without labeling a training set. VLMs can reach useful accuracy on many inspection tasks that previously required weeks of data collection and annotation.
• Alert triage: Security camera alerts are overwhelmingly false positives. A VLM reviewing flagged frames and providing a natural-language description of what triggered the alert (with a confidence assessment) dramatically reduces human review burden.
• Visual question answering for field operations: Maintenance technicians photographing equipment can get instant responses — "Is this valve handle in the open or closed position?" — without task-specific models trained on each piece of machinery.
• Document and label OCR + understanding: Florence-2 and similar grounding models handle structured extraction from visually complex documents (pharmaceutical labels, circuit boards, shipping manifests) where classical OCR pipelines struggled.

Grounding and spatial understanding

VLM grounding — the ability to localize objects or regions described in text — has matured significantly. Florence-2's phrase grounding and referring expression comprehension capabilities let you ask "where is the fire extinguisher?" and get a bounding box back, without any domain-specific training. Qwen-VL's spatial understanding has proven particularly strong in industrial contexts where precise localization matters. The practical constraint is that grounding accuracy degrades on small objects, cluttered scenes, and domain-specific terminology that wasn't well represented in pretraining — exactly the conditions that characterize many production deployments.

4. Multimodal Vision: Sensor Fusion in Practice

Pure RGB vision is blind to situations where the information you need isn't visible in the optical spectrum. Night scenes, fog, through-wall detection, material composition, 3D structure — all require additional sensing modalities. Production multimodal CV systems typically combine RGB with one or more of: depth (structured light, stereo, ToF), LiDAR, thermal infrared, radar, or audio. The architectural question is always where in the pipeline to fuse.

Early, late, and mid-level fusion

For most production surveillance, industrial, and logistics deployments, late fusion is the practical choice: it is easier to deploy, easier to debug, and allows each modality's model to be updated independently. Mid-level fusion with cross-attention is increasingly used where accuracy is paramount and engineering resources support the added complexity — autonomous vehicles (RGB + LiDAR), robotic surgery (RGB + depth + force), and high-end inspection systems.

Thermal + RGB: the security deployment reality

Thermal IR cameras are standard in perimeter security, data centers, and industrial facilities where night and adverse-weather detection is required. The practical fusion architecture in 2026 is almost always late fusion: a thermal detector running continuously (it is good at detection in darkness, immune to camouflage, privacy-safe), with an RGB model providing classification and appearance details when the thermal detector fires an alert. Fusing thermal and RGB at the feature level requires calibrated sensor alignment and a paired training dataset — feasible for a large security integrator, impractical for most production deployments.

5. Computer Vision at the Edge: Hailo, Jetson, and What Actually Ships

Edge CV deployment is where the gap between research and production is widest. Running a YOLOv8 detection model in a Python notebook is a one-hour exercise. Running it at 30 fps on a device that draws under 10 watts, embedded in a housing that reaches 70°C in direct sunlight, is a multi-week engineering project. Here is what the edge CV stack actually looks like in 2026.

The hardware landscape

NVIDIA Jetson (Orin NX, Orin Nano) remains the dominant platform for edge CV in research, robotics, and applications where you need flexibility. The full software stack — TensorRT, CUDA, NVIDIA's DeepStream for multi-camera inference pipelines — is mature and well-documented. The tradeoff is power (10–40W depending on module) and cost (the Orin NX starts above $200 at module cost). Most serious edge CV prototypes and mid-to-high-end products ship on Jetson.

Hailo, the Tel Aviv-based AI chip company, has carved out a significant position with its Hailo-8 and Hailo-8L NPUs. The Hailo-8 delivers 26 TOPS in a 2.5W envelope — performance per watt that Jetson cannot match. Hailo chips are embedded in Raspberry Pi AI Kit (making them accessible to a huge developer base), industrial cameras, and OEM products across smart cities, retail analytics, and automotive. The constraint is the Dataflow Compiler: you compile your model into a static dataflow graph, which is highly efficient but means dynamic control flow (the kind common in complex multi-stage pipelines) needs to be handled outside the chip. Working within this constraint is a learned skill — teams that have done it once understand which model architectures map cleanly and which require architectural changes.

Smartphone NPUs (Apple Neural Engine, Qualcomm Hexagon, MediaTek APU) are the most widely deployed CV inference hardware on earth, even though most CV engineers don't think of phones as edge devices. The Apple Neural Engine in the A17 Pro delivers approximately 35 TOPS; Qualcomm's Snapdragon 8 Elite is in the same range. Every major mobile CV use case — real-time portrait segmentation, scene understanding, augmented reality anchoring, document scanning — runs on these. The path to deployment is Core ML (Apple) or SNPE/QNN (Qualcomm), both of which accept ONNX as an interchange format.

The optimization stack: getting your model to run

The standard production path for edge CV model optimization follows a predictable sequence:

1. Export to ONNX: PyTorch → ONNX is now reliable for most standard architectures. Custom ops and dynamic shapes are the common failure points — address these first.
2. Quantization to INT8: Post-training quantization (PTQ) with a representative calibration dataset typically achieves 80–95% of FP32 accuracy with 2–4× speedup on INT8-capable hardware. For accuracy-sensitive applications (medical imaging, safety-critical detection), quantization-aware training (QAT) closes the remaining gap at the cost of retraining time.
3. TensorRT compilation: For Jetson, TensorRT compilation on top of the INT8 ONNX model delivers an additional 1.5–3× speedup through kernel fusion, layer optimization, and hardware-specific memory layout. The compiled engine is hardware-specific and must be regenerated for each device variant.
4. Hailo Dataflow Compiler: Hailo's toolchain ingests ONNX and performs the dataflow compilation step, including automatic quantization with the Hailo Model Zoo as a reference. Models with non-standard layers require manual layer implementation or architectural substitution.
5. Profiling and bottleneck analysis: NSight Systems (Jetson), Hailo Profiler, or Instruments (Apple) reveals whether bottlenecks are compute-bound or memory-bandwidth-bound — which determines whether further quantization, pruning, or architectural changes are the right lever.

The Israeli edge AI ecosystem

Israel is disproportionately significant in edge computer vision. Mobileye, acquired by Intel and then re-listed on NASDAQ, built the automotive CV industry — its EyeQ chips process camera, radar, and lidar data for driver assistance and autonomous driving in over 100 million vehicles. Mobileye's production deployment experience at automotive scale (functional safety certification, operating temperature range, multi-year software update constraints) informs what "production CV at the edge" actually means at its most demanding.

Hailo's Hailo-8L is specifically designed for cost-sensitive volume applications — embedded in smart cameras, retail analytics devices, and industrial inspection systems shipping in the hundreds of thousands. The company's partnerships with camera manufacturers like Axis and integration into the Raspberry Pi ecosystem have made it a first-choice option for integrators building new CV-enabled products.

Other Israeli companies active in edge CV include Edgify (federated learning for edge retail), Sievert (CV for industrial inspection), and a cluster of defense-adjacent computer vision companies that rarely appear in commercial press but drive significant IP development in tracking, thermal imaging, and sensor fusion.

6. Production Considerations: Monitoring, Annotation, and Privacy

Getting a CV model to pass offline evaluation metrics is the easy part. Keeping it performing in production — against a world that keeps changing — is the harder engineering discipline. Here are the considerations that most production CV teams learn the hard way.

Data drift in visual domains

Visual data drift is particularly insidious because it can be invisible to standard statistical tests applied to raw pixel values. Camera hardware upgrades (new lens, sensor, firmware update), seasonal changes in lighting (winter sun angles vs summer), facility changes (new paint color, rearranged fixtures), and even dust accumulation on a lens can shift the input distribution enough to meaningfully degrade model performance — without triggering alerts based on prediction confidence alone.

Effective visual drift monitoring combines: (1) tracking aggregate statistics of model intermediate activations (not just outputs), (2) periodic human review of randomly sampled hard negatives (frames where the model expressed low confidence), and (3) for production systems with ground truth, performance metric tracking via CUSUM or similar sequential change detection. For systems where ground truth is delayed or unavailable (like surveillance), proxy metrics — tracker ID switch rate, detection confidence distribution, inter-frame bounding box velocity distribution — serve as leading indicators of model degradation.

Annotation pipelines and active learning

Initial training data is rarely sufficient for production CV. Edge cases, rare object classes, new environmental conditions, and domain adaptation needs all require ongoing annotation. The production-grade annotation workflow for CV looks very different from a research lab's: it involves a labeling platform (Label Studio, Scale AI, CVAT) integrated with a model-in-the-loop, where the current model pre-annotates frames and human reviewers correct errors rather than labeling from scratch.

Active learning — selecting which frames are most informative to label, rather than labeling at random — is particularly high-value for CV where annotation is expensive and data volume is large. Core sampling strategies include uncertainty sampling (label frames where the model is least confident), diversity sampling (cover the input distribution rather than redundantly labeling similar frames), and error-driven sampling (prioritize frames from deployment scenarios where the model is known to underperform). The combination of these approaches typically achieves within 90% of full-dataset accuracy using 20–30% of the annotation budget — a meaningful economic advantage in production systems that require continuous retraining.

Privacy: face detection, blur, and regulatory compliance

Any CV deployment that processes video of people — which is most surveillance, retail, and public-space applications — faces privacy obligations that have become substantially more demanding in recent years. GDPR in Europe and Israel's Privacy Protection Regulations impose requirements on biometric data processing; the EU AI Act classifies real-time remote biometric identification in public spaces as prohibited AI (with narrow law enforcement exceptions) and general-purpose emotional recognition as high-risk.

The practical response in most commercial deployments is privacy-by-design: face blurring or replacement at the camera or edge device level, before any data leaves the physical premises. Production face blurring pipelines use lightweight face detectors (RetinaFace, YuNet) running at full frame rate on edge hardware, with Gaussian or pixelation blur applied to detected bounding boxes before the frame is forwarded to any downstream model or storage. The latency overhead is typically under 5ms on INT8-quantized detectors. For analytics use cases that need body-level tracking without facial identification, skeleton pose estimation (replacing bounding boxes with keypoint representations) has become the privacy-compliant alternative to person tracking with appearance features.

Computer vision in production is a discipline of tradeoffs: accuracy versus latency, generality versus optimization, rich sensor fusion versus engineering complexity, capability versus privacy compliance. The field has matured to the point where the algorithm questions are largely answerable — ByteTrack for tracking, RAFT variants for flow, VLMs for open-ended understanding, INT8 quantization for edge deployment. The questions that separate successful production deployments from failed ones are operational: how do you detect when your model has drifted, how do you keep your annotation pipeline current with your deployment environment, and how do you design for the privacy obligations that come with any camera-based system. Getting those right is what "making AI work in production" actually means in computer vision.