A 32.8mW 60fps cortical vision processor for spatio-temporal action recognition

In this paper, we propose a 60fps cortical vision processor modeling a hierarchical object classification model (HMAX) based on spatio-temporality in video stream. It is hard to implement a real-time hardware for HMAX operation due to its high computational cost from 2-D template matching. Three components are proposed with our improved algorithms. Class Refinement Structure (CRS) dramatically reduces a dimension of HMAX descriptors by 97.01% compared to the previous works by exploiting spatio-temporal features in video action recognition. Spatio-Temporal Memory Structure (STMS) adopts spatially adaptive window technique, and it reduces the required on-chip data bandwidth and computations per a template in S2 stage. In addition, a dual image buffer structure also reduces the required off-chip network bandwidth for processing complex hierarchical stages and numerous image spaces. As a result, the 10.8 GOPS cortical vision processor implemented in 0.13μm CMOS process achieves 60frames/sec performances for 256×256 video inputs at 200MHz operating frequency.