Coupled Primary and Secondary Transform for Next Generation Video Coding

The discrete cosine transform type II can efficiently approximate the Karhunen-Loeve transform under the first-order stationary Markov condition. However, the highly dynamic characteristics of natural images will not always follow the first-order stationary Markov condition. It is well known that multi-core transforms and non-separable transforms capture diversified and directional texture patterns more efficiently. And a combination of enhanced multiple transform (EMT) and non-separable secondary transform (NSST) are provided in the reference software of the next generation video coding standard to solve this problem. However, the current method of combining the EMT and NSST may lead to quite significant encoder complexity increase, which makes the video codec rather impractical for real applications. Therefore, in this paper, we investigate the interactions between EMT and NSST, and propose a coupled primary and secondary transform to simplify the combination to obtain a better trade-off between the performance and the encoder complexity. With the proposed method, the transform for the Luma and Chroma components is also unified for a consistent design as an additional benefit. We implement the proposed transform on top of the Next software, which has been proposed for the next generation video coding standard. The experimental results demonstrate that the proposed algorithm can provide significant time reduction while keeping the majority of the performance.