Octopus: Context-Aware CNN Inference for IoT Applications

Modern convolutional neural networks (CNNs) in computer vision are trained on a large number of images from numerous categories to form rich discriminative feature extractors. Inference using such models on resource-constrained Internet-of-Things (IoT) platforms poses a challenge and an opportunity. Having limited computation, storage, and energy budgets, most IoT platforms are not capable of hosting such compute intensive models. However, typical IoT applications demand detection of a relatively small number of categories, albeit the specific categories of interest may change at runtime as the context evolves dynamically. In this letter, we take advantage of the opportunity to address the challenge. Specifically, we develop a novel transformation to the architecture of a given CNN, so that the majority of the inference workload is allocated to class-specific disjoint branches, which can be dynamically executed or skipped, based on the context, to fulfill the application requirements. Experiments demonstrate that our approach preserves the classification accuracy for the classes of interest, while proportionally decreasing the model complexity and inference workload.