Segmentation of video with dynamic background is an important research topic in image analysis and computer vision domains. In this paper, we present a novel recursive Bayesian learning-based method for the efficient and accurate segmentation of video with dynamic background. In the algorithm, each frame pixel is represented as the layered normal distributions which correspond to different background contents in the scene. The layers are associated with a confident term and only the layers satisfy the given confidence which will be updated via the recursive Bayesian estimation. This makes learning of background motion trajectories more accurate and efficient. To improve the segmentation quality, the coarse foreground is obtained via simple background subtraction first. Then, a local texture correlation operator is introduced to fill the vacancies and remove the fractional false foreground regions. Extensive experiments on a variety of public video datasets and comparisons with some classical and recent algorithms are used to demonstrate its improvements in both segmentation accuracy and efficiency.