Research deep-dive / Architecture

Edge vs cloud latency for video AI

Edge and cloud differ less in average latency than in worst-case latency. A local inference path typically completes in tens of milliseconds with low variance, while a cloud path adds a physics floor set by distance plus queueing whose tail is far worse than its mean. Whether that matters depends entirely on whether the output drives an action.

Reviewed by Dr. Vivek Gujar, Chief Strategy Officer  ·  31 August 2026  ·  13 min read

The latency argument for edge computing is usually made badly. It is asserted that cloud is slow and edge is fast, a number like 100 milliseconds is quoted without a source, and the discussion moves on. That framing is unhelpful in both directions: it oversells edge for workloads that do not need it, and it obscures the cases where the difference is genuinely decisive.

This deep-dive decomposes the latency budget term by term, establishes what the physics permits, and then makes the argument that matters more than the mean: variance. It also identifies where cloud inference is entirely appropriate, which is a larger set of cases than edge vendors typically concede.

The full budget, glass to glass

The number that matters operationally is glass to glass latency: the elapsed time from light striking the sensor to a decision being available for action. Inference time, the figure most commonly quoted, is one term among seven.

Two observations follow immediately, and both are usually skipped.

The first is that encode latency is paid identically in both architectures. Video encoders buffer frames to compress efficiently, and the group of pictures structure means some frames cannot be emitted until later ones have been processed. This is a real cost incurred before anything reaches a network, and no architectural choice removes it.

The second is that cloud inference is frequently faster than edge inference in the compute step itself. A datacentre accelerator is far more capable than a fanless appliance in a panel. Edge does not win on compute. It wins by deleting the transport and queueing terms entirely.

The physics floor

Some of the cloud penalty is not an engineering problem and cannot be optimised away. Light travels through single mode fibre at roughly two thirds of its vacuum speed, about 200,000 km per second, and real fibre routes are longer than great circle distances because cables follow coastlines, existing rights of way and landing stations.

The corollary is often overlooked by edge advocates: a nearby region changes the picture substantially. At a 2 millisecond floor to a Mumbai region, the physics term is negligible and the argument has to be made on other grounds. Cloud region proximity is a real architectural variable, not a rounding error.

Why the mean is the wrong number

The decisive argument is not average latency. It is variance, and the canonical treatment is Dean and Barroso's 2013 paper in Communications of the ACM.

Their central observation is that latency variability in individual components is amplified at the service level. They give the example of a system where each server typically responds in 10 milliseconds but has a 99th percentile of one second: handled on a single such server, one request in a hundred is slow.

The amplification arrives with fan-out. When a request touches 100 backends and each has a 1 in 100 chance of being slow, roughly 63 percent of user requests encounter at least one slow backend. In the paper's more extreme case, with only 1 request in 10,000 slow per server and a fan-out of 2,000 servers, close to one in five user requests still takes a second or more.

Measurements from a real Google service make the point concretely. The 99th percentile latency for a single random request measured at the root was 10 milliseconds. The 99th percentile for all requests in the fan-out to finish was 140 milliseconds, and for 95 percent of requests finishing it was 70 milliseconds.

Applied to video analytics, the relevance is that a cloud inference path is not one hop. It is a load balancer, an ingress gateway, an authentication service, a queue, a model server and a response path, each with its own tail. A path advertised at 40 milliseconds mean can have a 99th percentile several times higher, and it is the 99th percentile that determines whether a barrier lifts before the driver reverses.

Dean and Barroso also describe mitigations, and they are instructive because of what they require. Hedged requests, where a duplicate is issued after a short delay, cut the 99.9th percentile in one Google benchmark from 1,800 milliseconds to 74 while adding about 2 percent more requests. That works when you control both ends and can afford duplicate work. In a video pipeline where the payload is a frame rather than a key lookup, duplicating requests to cut the tail means sending the frame twice, which reintroduces the bandwidth problem the cloud architecture was chosen to avoid.

The 5G argument, examined honestly

A common response is that 5G removes the problem. The figure usually cited is 1 millisecond, and it is a real 3GPP target: the URLLC requirement specifies a hard latency of one millisecond over the air interface with system reliability of 99.999 percent, with user plane targets of 0.5 milliseconds in each direction.

The honest reading is that 5G with multi-access edge computing moves computation closer to the radio and can remove a large fraction of network delay. That is a genuine improvement and it is also, structurally, an edge architecture. It does not make a distant region behave like a local one; it makes the region less distant.

Which tasks actually have a latency budget

Most video analytics tasks have budgets measured in minutes, and treating them as latency-critical wastes money. The discipline is to work backwards from the physical process: how much time exists between the earliest moment the event is detectable and the last moment an intervention is useful.

Availability is usually the stronger argument

Latency dominates the discussion, but for most Indian industrial and commercial sites the more consequential difference is availability. A cloud analytics deployment with a single site broadband connection has no analytics at all during any outage, planned or otherwise.

For security applications this is more than an inconvenience, because a link failure and a security event are not independent. Anyone deliberately interfering with a site has an obvious interest in the connection, and monsoon-season power and link interruptions cluster with exactly the conditions under which perimeter monitoring matters. A local processing path continues to detect and alert with the wider internet down, provided the alert has a local destination.

This is also the argument that survives cloud regions getting closer. Region proximity fixes latency. It does not fix the site link.

The hybrid pattern

Framing the choice as edge or cloud is a false binary, and the pattern most deployments converge on uses both according to where each workload's budget and data volume point.

This split resolves several problems at once. The time-critical path gets a bounded tail. The analytical path gets the compute and the convenience of a managed service. Bandwidth collapses because events rather than streams cross the link, an effect quantified in our companion piece on edge AI. And the data residency question becomes tractable, because the footage never leaves the site even though the aggregate does.

How to make the decision

Four questions, in order:

1. What is the physical budget? Derive it from the process, not from the technology, and express it as a 99th percentile rather than a mean.
2. What is the physics floor to the nearest region? If the floor already consumes a large share of the budget, the decision is made. If it is 2 milliseconds, keep going.
3. What is the tail, not the mean? Measure it over a working week including peak hours, not for ten minutes on a quiet afternoon.
4. What happens when the link is down? If the answer is unacceptable for any period, local processing is required regardless of what the latency analysis concluded.

Question four decides more real deployments than questions one to three combined, which is why the latency argument, though genuine, is not usually the argument that should be leading.

Frequently asked questions

References

1. Dean, J., and Barroso, L. A., The Tail at Scale, Communications of the ACM, 56(2), 74 to 80, 2013. DOI 10.1145/2408776.2408794.
2. 3GPP TR 38.913, requirements for next generation access technologies, URLLC user plane and reliability targets.
3. Ultra-Reliable Low-Latency in 5G: A Close Reality or a Distant Goal?, HotNets 2024. On the gap between URLLC specification and practical implementation.
4. 5G Americas, Ultra-Reliable Low-Latency Communication, white paper. End-to-end budget decomposition and the air interface share.
5. Ultra-low-latency services in 5G systems: a perspective from 3GPP standards, ETRI Journal, 2021. On end-to-end latency of 1 to 10 ms and the role of multi-access edge computing.
6. Nguyen, M., et al., The Tail at Scale: How to Predict It?, USENIX HotCloud 2016.
7. Cisco, Three Services from 5G: Latency and Reliability, on multi-access edge computing removing backhaul delay.