Adaptive Smoothing of Non-Rectangular Prediction Block Edges in the Wedge Mode of AVM

This work introduces an additional boundary for extended wedge mode in the reference software of AVM. Wedge mode, introduced in AV1 and modified later, employs a non-rectangular partitioning mode to combine two prediction signals. However, the available wedge masks might not be sufficient to handle diverse video content. In this work, an adaptive boundary selection scheme to construct the wedge masks is proposed. A set of two boundaries are used for this purpose. Additionally, three signaling schemes are presented, with different levels of complexity, and one low-complexity design which limits the adaptivity to smaller block sizes while maintaining similar performance. Experimental results using AVM common testing conditions showed promising gains. In random access configuration, class A5 demonstrates 0.89% PSNR-Y BD rate gain, while classes A1, B1, and A3 show approximately 0.08% PSNR-Y BD rate gain.