This paper illustrates multiple levels of boundary for video segmentation from coarse to fine granularity, which include highlights, attentive visual change, and game status change. The boundaries between video shots are commonly known as scene change and the action of segmentation a video sequence into multiple shots is called scene change detection. However, different applications are suitable for different fine granularities of the shot boundary. In this paper, we take the content semantic into consideration to segment the video. The experimental results show the efficiency of the proposed method for sports programs.