A smart audio/video mixing method based on main speaker decision for a software-based multipoint control unit

In this paper, we present a Smart Audio/Video Mixing method (SmartAVM) for a software-based multipoint control unit (SW-based MCU) to provide an immersive video conferencing service. The SmartAVM determines the mixing mode to one of the true mixing, equal mixing, and main speaker decision-based mixing modes, according to the number of clients (i.e., human users) connected to the SW-based MCU. Then, audio/video (A/V) signals from all clients are mixed using the different weighting factors according to the mixing mode. To show the effectiveness of SmartAVM, we implemented an SW-based MCU using OPUS and VP8 as the audio and video coders, respectively. Results show that the SmartAVM correctly determined the proper mixing mode and mixed A/V signals according to the determined mixing mode.