Visual sensing plays an essential role in intelligent vehicles. With the aid of visual sensors, driver assistance systems can alert the driver to dangerous situations or actions (such as swerving out of the lane or disregarding traffic signs or lights), or even independently take control of the vehicle. Currently, research on vehicle vision systems is receiving continuous global interest.This paper discusses some future directions for this field based on trends we observed at the 2008 IEEE Conference on Intelligent Transportation Systems (ITSC 08) and the 2009 IEEE Intelligent Vehicles Symposium (IVS 09). Our aim is to attract researchers to several critical questions that are important yet difficult. Solving these problems could be of great benefit to both academia and the vehicle industries.