Machine learning-based video analytics pipelines have become the de facto solution for automated and accurate video processing. However, deploying video analytics pipelines has become increasingly prohibitive due to the growing number of camera feeds, more complex and specialized vision models, and the broadening range of applications and use cases. Practically supporting this rising demand necessitates the efficient utilization of underlying resources (e.g., heterogeneous hardware, storage, network bandwidth). My research focuses on designing automated and general resource optimization techniques and building systems to practically incorporate these techniques into existing video analytics pipelines. My work has targeted 3 key operational scenarios: querying large-scale video datasets in the cloud, real-time analytics in resource-constrained edge settings, and privacy-preserving video analytics queries.

In today's talk, I will discuss one of these projects, Gemel, which aims to reduce GPU memory pressure in edge video analytics deployments. We observe that edge-box GPUs lack the memory needed to concurrently house the growing number of (increasingly complex) models for real-time inference. We present model merging, a new memory management technique that exploits architectural similarities between edge vision models by judiciously sharing their layers (including weights) to reduce workload memory costs and swapping delays. Gemel efficiently integrates merging into existing pipelines by (1) leveraging several guiding observations about per-model memory usage and inter-layer dependencies to quickly identify fruitful and accuracy-preserving merging configurations, and (2) altering edge inference schedules to maximize merging benefits.