In this paper, we propose an adaptive scalable layer selection policy for video streaming over wireless networks by jointly considering the wireless channel conditions and scalable video coding. Our goal is to design a scheme which could provide both long-term smooth and high play-back quality for each user, while maintaining the fairness among multi-users. We first design an adaptive long-term layer selection scheme which determines the number of layers each user would receive in order to maximize the total video quality of all streams while avoiding high variation in the number of video layers. We then employ a scheduling algorithm for slot allocation considering both current channel capacity and deadline. We compare our proposed policies with some reference schemes and simulation results show the effectiveness of our proposed scheme.