As the high growth of the number of vehicles, the traffic accidents are becoming more and more serious in recent years. In order to avoid the drivers being in danger, an intelligent vision-based system should focus on the image contents of the front the camera setting under the rear-view mirror in the vehicle. In this paper, we present a functional-link-based neuro-fuzzy network (FLNFN) structure for lane detection and departure warning system application. The proposed FLNFN model uses a functional link neural network (FLNN) to the consequent part of the fuzzy rules. This study uses orthogonal polynomials and linearly independent functions in a functional expansion of the FLNN. Thus, the consequent part of the proposed FLNFN model is a nonlinear combination of input variables. An online learning algorithm, which consists of structure learning and parameter learning, is also presented. The structure learning depends on the entropy measure to determine the number of fuzzy rules. The parameter learning, based on the gradient descent method, can adjust the shape of the membership function and the corresponding weights of the FLNN. The lane detection method and the departure warning system proposed in this paper have been successfully evaluated on a PC platform of 3.2-GHz CPU, where the average frame-rate is up to 30fps.