A CNN-Based Super-Resolution Processor With Short-Term Caching for Real-Time UHD Upscaling

Super-resolution (SR) tasks, involving the restoration of low-resolution images to high-resolution images, are expected to handle larger images in the near future. This paper proposes the short-term caching (STC) layer fusion to address the increase in cache memory size for image size expansion. The proposed STC layer fusion requires only 0.2% of the memory size compared to the previous method by discarding the overlap data that must be maintained for a long time and recalculating the discarded data later. In addition, the selective-operated dual accumulator (SODA) is proposed to apply a vertically long output patch that minimizes the additional recalculation of the STC layer fusion with resolving the memory access problem. Thus, the number of processing elements and memory bandwidth are reduced by 33.6% and 35.6%, respectively. The proposed SR processor including the proposed STC layer fusion and SODA is fabricated on a 28nm CMOS process, and the die and core area occupy 12.96mm2 and 3.45mm2, respectively. The SR task achieves
$\times 2$
upscaling to UHD images at 60 fps, achieving a maximum SR throughput of 497.7 Mpixels/s. Compared to previous approaches, the proposed SR processor reduces the on-chip memory size by more than 95% and achieves 4.3 times the SR throughput.