HaDP: Edge Inference via Hardness-Aware Data Partitioning

The BranchyNet framework for edge inference has been widely studied and shown a huge improvement in latency. However, the improved latency comes at the cost of inference accuracy, which may drop if a large portion of data samples exits at the side branches. To tackle this issue, we integrate negative correlation learning into BranchyNet in this study. In particular, we treat all branches as ensemble members and jointly train them with a diversity-enhanced loss function. For that purpose, we propose a hardness-aware data partitioning scheme to handle easy samples using the class-specific thresholds and tackle hard data by the diversity-enhanced ensemble. Experiments show that the proposed approach can reduce the data processed by the main branch by 40% while maintaining the inference accuracy. Furthermore, the analysis also shows a trade-off between latency and accuracy in our ensemble setting.