Research deep-dive / Edge infrastructure

Edge compute sizing for video AI workloads

Edge compute sizing means deriving hardware requirements from the actual frame and inference budget of a workload rather than from camera count or a datasheet TOPS figure. The two numbers that decide it are decode load and achieved utilisation, and both are routinely omitted from vendor sizing guidance.

Reviewed by Dr. Vivek Gujar, Chief Strategy Officer  ·  31 August 2026  ·  14 min read

Sizing an edge appliance for video analytics is usually done with a rule of thumb: so many cameras per device. It is a convenient rule and it is wrong often enough to be dangerous, because camera count is not the workload. Two sites with sixty four cameras each can differ by more than an order of magnitude in the compute they require, depending on what is being asked of the video.

This deep-dive derives the sizing from first principles. It covers the frame budget, the decode term that vendor guidance almost always omits, what quantization actually costs, why a datasheet TOPS figure predicts very little, and how to read a throughput claim. The arithmetic throughout is worked openly so it can be applied to a different estate.

The four stages, and where the time goes

Every analytics pipeline performs four operations on every frame it processes. Sizing exercises tend to consider only the third.

The consequence is that an appliance can sit at 30 percent accelerator utilisation and still drop frames, because the video decode engine is saturated. Diagnosed from accelerator utilisation alone, this looks like idle hardware failing for no reason. Measure the two independently or the bottleneck will be invisible.

The frame budget

The correct starting point is not cameras but frames, and not frames but inferences. The workload is the sum across cameras of frame rate multiplied by the number of models applied to that camera:

These two sums have different shapes, and that difference is where sizing errors originate. A camera sampled at 5 fps on a sub stream contributes little to both. A camera running plate recognition at full rate on a main stream contributes moderately to the first and enormously to the second.

The design response is to distribute by workload rather than by count, or to separate the plate cameras onto their own appliance so their decode demand does not starve the rest.

Why TOPS predicts so little

Accelerator datasheets lead with a TOPS figure, meaning trillions of operations per second. It is a peak arithmetic rate measured under conditions chosen to maximise it. Whether a workload approaches it depends on a property of the workload, not of the chip.

The governing framework is the roofline model, introduced by Williams, Waterman and Patterson. It bounds attainable performance as the lesser of two ceilings: the peak compute rate, and the product of memory bandwidth and arithmetic intensity, which is the number of operations performed per byte moved from memory.

Below the ridge point a workload is memory-bound and the compute ceiling is irrelevant. On an NVIDIA A100, with roughly 312 TFLOPS against 2 TB/s of bandwidth, the ridge sits near 156 operations per byte. Any workload below that intensity cannot reach the advertised compute rate no matter how the code is written.

There is a second problem with datasheet figures, and it applies to the bandwidth number as well as the compute number. A study characterising neural network inference on mobile system-on-chips found that although the specification claimed a peak memory bandwidth of 14.9 GB/s, measured sustainable bandwidth per component was 3.44 GB/s, 0.49 GB/s and 6.15 GB/s for the two CPU clusters and the GPU respectively.

A practical translation of all this into a sizing rule:

What quantization actually costs

Edge deployment almost always involves reducing numerical precision, usually to INT8, to fit memory and accelerate inference. The saving is real and so is the cost, and the cost is frequently understated.

A 2025 empirical study evaluated YOLO models across five scales, from nano to extra-large, in FP32, FP16, dynamic UINT8 and static INT8, benchmarked on COCO under seven degradation conditions. It found that static INT8 TensorRT engines delivered speedups of roughly 1.5 to 3.3 times with a moderate accuracy drop of about 3 to 7 percent mAP50-95 on clean data.

That is the well-behaved outcome, and it is a reasonable planning assumption. Two findings temper it.

The first is that quantization can fail badly rather than gracefully. An NVIDIA TensorRT issue report documents a YOLOv5s model whose COCO mAP fell from 0.362 to 0.054 under INT8 conversion using the default graph, recovering to 0.316 only once a custom plugin handled a specific activation function correctly. The difference between a 13 percent relative accuracy loss and an 85 percent one was an implementation detail in the conversion toolchain, not a property of the model.

The second is that model capacity affects resilience. Work on vulnerable road user detection reports a teacher model suffering catastrophic degradation under INT8 at 23 percent mAP lost, while a knowledge-distilled student retained accuracy far better at 5.6 percent lost, in a model nearly four times smaller. Smaller and better-prepared is not always worse.

The same 2025 study also tested a degradation-aware calibration strategy, exposing the calibration process to a mix of clean and synthetically degraded images. It did not produce consistent improvements in robustness across most models and degradations. Calibration set design is not a reliable substitute for measuring the quantized model on real footage from the deployment.

The governing rule: any accuracy figure quoted by a vendor must be the figure for the quantized model on the target device, not the full precision model on a server. These are different numbers and the difference is the one that will show up in your alert log. Accuracy loss also compounds with the base rate problem, since a few points of precision lost on a rare-event detector can change alert volume substantially.

Reading a throughput claim

Vendor throughput numbers are frequently incomparable with each other because they measure different things. The MLPerf Inference benchmark, developed by MLCommons, exists partly to impose discipline here, and its structure is a useful lens even when a vendor has not submitted results.

MLPerf defines four scenarios. Single-stream sends one query at a time and reports 90th percentile latency. Multistream sends queries of eight samples and reports 99th percentile latency, which the benchmark authors describe as reflecting industrial automation and remote sensing tasks such as vehicles analysing several cameras at once. Server generates queries on a Poisson distribution under a quality of service constraint. Offline submits everything at once and reports raw throughput with latency unconstrained. Edge submissions use the first, second and fourth.

NVIDIA's own commentary on the benchmark makes the point plainly, cautioning that comparisons must be kept like for like and that one platform's offline result should not be set against another's multi-stream figure.

MLPerf also imposes an accuracy floor, requiring edge submissions to reach 99 percent of the reference model's accuracy, or 99.9 percent for high-accuracy variants. This is worth borrowing. A throughput claim without an accompanying accuracy constraint can always be met by degrading the model, and frequently is.

Headroom, thermals and failure domain

Three factors turn a correct calculation into a wrong deployment.

Scene complexity is variable

Detection cost varies with what is in the frame. A car park at 03:00 and the same car park at shift change do not cost the same, because postprocessing scales with the number of detections. Sizing against average load produces a system that fails precisely when the site is busiest, which is when it matters.

Thermal behaviour differs from bench behaviour

A sixty second benchmark completes before a device reaches steady-state temperature. Edge appliances in Indian deployments frequently sit in unventilated panels in ambient conditions well above a laboratory. Sustained clocks under those conditions can be materially below burst clocks. Any sizing test that has not run for at least an hour at the target ambient temperature has measured burst performance and called it sustained performance.

Recovery bursts

After a network interruption or a service restart, queued work arrives at once. A system sized with no margin will not catch up, and will instead shed frames for an extended period after every transient fault.

On headroom, a defensible target is to run at no more than 60 to 70 percent of measured sustained capacity. That margin is not conservatism for its own sake. It absorbs scene complexity variation, thermal derating, recovery bursts, and the near certainty that a future model version will cost more than the one it replaces.

One appliance or several

Given a total workload, it can be met with one large device or several smaller ones. Several smaller units generally fail better: one failing degrades coverage rather than eliminating it, cooling is easier, and capacity can be added incrementally as models are added. The cost is more devices to monitor, though fleet monitoring is required in either case.

Where an estate contains a small number of very expensive streams, as in the worked example above, isolating them on their own device is usually worth doing regardless of the total, because it prevents one workload class from starving another.

For the upstream decisions that determine the frame budget in the first place, see our companion piece on RTSP stream optimization. Reducing what the appliance is asked to decode is nearly always cheaper than buying a larger appliance, and the two exercises should be done together rather than in sequence.

Frequently asked questions

References

1. Williams, S., Waterman, A., and Patterson, D., Roofline: An Insightful Visual Performance Model for Multicore Architectures, Communications of the ACM, 52(4), 2009.
2. Reddi, V. J., et al., MLPerf Inference Benchmark, MLCommons, arXiv:1911.02549, 2019.
3. MLCommons, MLPerf Inference Rules, scenario definitions and latency constraints.
4. Quantization Robustness to Input Degradations for Object Detection, arXiv:2508.19600, 2025. Static INT8 speedup and mAP50-95 findings.
5. NVIDIA TensorRT, Quantization Error Analysis on Object Detection, issue 1114. YOLOv5s INT8 mAP collapse and recovery.
6. Neural Network Inference on Mobile SoCs, arXiv:1908.11450. Specified versus measured memory bandwidth.
7. NVIDIA, MLPerf Inference: NVIDIA Innovations Bring Leading Performance, on like-for-like scenario comparison.