Modern automated video analysis systems consist of large networks of heterogeneous sensors. These systems must extract, integrate and present relevant information from the sensors in real-time. This paper addresses some of the major challenges such systems face: efficient video processing for high-resolution sensors; data fusion across multiple modalities; robustness to changing environmental conditions and video processing errors; and intuitive user interfaces for visualization and analysis. The paper discusses enabling technologies to overcome these challenges and presents a case study of a wide area video analysis system deployed at a port in the state of Florida, USA. The components of the system are also detailed and justified using quantitative and qualitative results.