Real-time video analytics uses AI to watch live camera feeds and flag or act on events within seconds. In 2026 it works in two layers: fast detection models run at the edge, next to the cameras, and vision language models (VLMs) then check alerts, answer questions and summarise footage. Processing at the edge keeps latency and bandwidth low and keeps raw video on site.
Updated October 2026
What changed in 2026
- VLMs became the second layer. NVIDIA's Video Search and Summarization (VSS) blueprint reached version 3 in May 2026. A detector raises an alert, then a VLM verifies it to cut false positives (NVIDIA).
- Edge boxes can run VLMs. In NVIDIA's benchmark on DGX Spark and Jetson AGX Thor hardware, VLM alert verification handled up to 14 concurrent streams with about 0.89 seconds of verification latency (same source).
- Video platforms added language. Milestone launched a VLM built on NVIDIA Cosmos Reason in Dec 2025, trained on 75,000 hours of video, for summaries and search inside XProtect (Milestone).
- EU rules were reset. The Digital Omnibus (Regulation (EU) 2026/1744) moved high-risk AI duties, including biometric systems, from 2 Aug 2026 to 2 Dec 2027 (Orrick). The ban on real-time remote biometric identification in public spaces has applied since Feb 2025 (Latham & Watkins).
What is real-time video analytics?
It is software that analyses video as it is captured, not hours later. It turns pixels into events: "person in restricted zone", "pallet blocking fire exit", "queue longer than eight people".
Three things make it "real time":
- Low delay. The time from frame to alert is measured in milliseconds to a few seconds.
- Continuous. It runs on every stream, all the time.
- Actionable. The output is an alert, a count or an action, not just a recording.
See computer vision solutions for the wider field.
How it works: from camera to decision
- Capture. IP cameras stream video, usually over RTSP.
- Decode and sample. Frames are decoded on a GPU and sampled at the rate the task needs.
- Detect and track. A fast model finds objects and follows them across frames.
- Apply rules. Zones, line crossings, dwell times and counts turn detections into events.
- Verify and explain. A VLM looks at the clip and confirms or rejects the alert, in words.
- Act. The system notifies a person, opens a ticket or triggers a workflow, within set limits.
- Learn. Confirmed and rejected alerts feed the next model update.
Steps 1 to 4 have existed for years. Step 5 is the main change since 2024.
Rule-based, AI and agentic video analytics compared
| Rule-based (motion) | AI detection (deep learning) | VLM and agent layer (2026) | |
|---|---|---|---|
| What it sees | Pixel change | Trained object classes | Scenes, actions, context |
| Setup | Draw zones, set thresholds | Train or pick a model | Write a prompt or policy |
| False alarms | High (shadows, rain) | Lower | Lower again, via verification |
| Ask it a question? | No | No | Yes, in plain language |
| Compute | Very low | Moderate, edge GPU | Higher, edge GPU or server |
| Typical era | 1990s to 2010s | Late 2010s onward | 2025 onward |
Most real systems in 2026 combine the last two columns. Detection is cheap and fast, so it runs on every frame (the same machine vision models used on production lines). The VLM is slower, so it runs only on clips that matter.
Why the edge, not the cloud
The 2024 version of this post made the case for edge processing. It still holds, for three reasons:
- Latency. A round trip to the cloud adds delay and depends on the network. Safety alerts cannot wait for a slow link.
- Bandwidth. Many high-resolution streams are costly to upload around the clock. Sending events and short clips is far cheaper.
- Privacy. Raw video that stays on site is easier to protect and to keep within data rules.
The cloud still has a role: training models, long-term search across sites and dashboards. A hybrid design is normal. See NexaStack Edge AI.
Examples of real-time video analytics
| Sector | What it watches for | Typical action |
|---|---|---|
| Manufacturing | Missed assembly steps, PPE, people near machines | Stop line or alert supervisor |
| Warehousing | Blocked exits, forklift near-misses, dock status | Alert, log safety event |
| Retail | Queue length, empty shelves | Open a till, restock |
| Cities and traffic | Accidents, flooding, wrong-way vehicles | Dispatch, change signals |
| Airports and transport | Crowding, left items, gate delays | Redeploy staff |
| Energy and utilities | Leaks, smoke, intrusions at remote sites | Alarm, dispatch crew |
Named examples from NVIDIA's VSS launch: Pegatron checks circuit-board assembly steps, and Linker Vision supports 12 city departments in Kaohsiung, Taiwan (NVIDIA, May 2025). Results quoted there are vendor-reported.
Can AI analyse a video? Yes, and now in words
A VLM can watch a clip and describe it, answer questions about it or check it against a rule. That changes how people use video:
- Search. "Show forklifts in aisle 4 after 10 pm" instead of scrubbing hours of footage.
- Summaries. NVIDIA says VSS can summarise an hour of video in under a minute (NVIDIA).
- Verification. The VLM confirms whether a detection is real before a person is paged.
VLMs can still be wrong. Keep a person in the loop for any action with safety, legal or staff consequences.
Privacy and the law
- EU AI Act. Real-time remote biometric identification in publicly accessible spaces for law enforcement is banned, with narrow exceptions, since 2 Feb 2025. Other biometric uses are high-risk, with duties from 2 Dec 2027 (Orrick).
- GDPR. Video of identifiable people is personal data. You need a lawful basis, signage, retention limits and, often, a data protection impact assessment.
- Design choices that help. Detect objects and behaviour, not identities. Blur faces by default. Keep raw video on site and send only events.
How to roll out real-time video analytics
- Start with one use case and a number. For example, cut false intrusion alarms or count forklift near-misses.
- Audit cameras. Check angle, resolution, lighting and network for that use case.
- Pick edge hardware for the load. Count streams, frame rates and whether you need a VLM on site.
- Pilot on a few cameras. Measure precision and recall against human review.
- Set the action rules. Decide what the system may do alone and what needs a person.
- Plan updates. Models drift as seasons, layouts and uniforms change. Version and roll out updates in stages.
- Scale by site. Copy the working setup, then tune per site.
Market size
MarketsandMarkets valued the video analytics market at $8.3 billion in 2023 and forecasts $22.6 billion by 2028, a 22.3% annual growth rate (GlobeNewswire, Aug 2025). Treat any market forecast as an estimate.
Where it's heading: 2026 → 2030
| Period | What it looks like |
|---|---|
| Around 2020 | Deep learning detectors replace most motion rules. Fixed object classes. High false-alarm rates; many systems process video on central servers. |
| 2026 (now) | Detection at the edge plus VLM verification and search (NVIDIA VSS 3, Milestone VLM). EU ban on real-time remote biometric ID in force since Feb 2025. |
| By 2028 (expected) | EU high-risk duties for biometric systems apply from 2 Dec 2027. Market forecast at $22.6 billion (MarketsandMarkets). VLMs expected to run on more edge boxes and cameras. |
| By 2030 (expected) | High-risk deadline for public-authority systems on 2 Aug 2030 (Orrick). Video agents expected to act within approved playbooks, with every action logged. |
Where NexaStack fits
Real-time video analytics is a fleet problem: many cameras, many edge boxes and models that need regular updates. NexaStack deploys, updates and monitors AI models and agents across edge devices. Teams can push a new detector or VLM to one site, check its alert accuracy, then roll it out or back. Each alert and action is logged against the model version and policy that produced it, so decisions stay governed and auditable.
Frequently asked questions
What is real-time video analytics?
It is the use of AI to analyse live video as it is captured and to raise alerts or take actions within seconds. Typical outputs are detections, counts, zone breaches and summaries.
Is there an AI that can analyse a video?
Yes. Detection models find and track objects. Vision language models, such as those used in NVIDIA's VSS blueprint and Milestone's XProtect summaries, can describe a clip, answer questions about it and check alerts.
What are some examples of video analytics?
Safety alerts when a person enters a machine zone, queue counts in retail, blocked-exit checks in warehouses, traffic incident detection in cities and intrusion alerts at remote energy sites.
Why process video analytics at the edge?
Edge processing cuts delay, saves bandwidth and keeps raw video on site. The cloud is still used for training models and searching across many sites.
Is AI video analytics legal under the EU AI Act?
Most uses are legal with safeguards. Real-time remote biometric identification in public spaces for law enforcement is banned, with narrow exceptions. Other biometric systems are high-risk, with obligations from 2 Dec 2027. GDPR applies to any footage of identifiable people. Take legal advice for your case.
Turn your camera feeds into governed decisions
Pick one site and one use case, and we will scope how NexaStack runs detection and VLM verification on your edge hardware. Book a 30-minute edge video analytics pilot call →
