Collaborative defense as a pervasive service Architectural insights and validation methodologies of a trial deployment

Network defense is an elusive art. The arsenal to defend our devices from attack is constantly lagging behind the latest methods used by attackers to break into them and subsequently into our networks. To counteract this trend, we developed a distributed, scalable approach that harnesses the power of collaborative end-host detectors or sensors. Simulation results reveal order of magnitude improvements over stand-alone detectors in the accuracy of detection (fewer false alarms) and in the quality of detection (the ability to capture stealthy anomalies that would otherwise go undetected). Although these results arise out of a proof of concept in the arena of botnet detection in an enterprise network, they have broader applicability to the area of network self-manageability of pervasive computing devices. To test the efficacy of these ideas further, Intel Corporation partnered with British Telecommunications plc to launch a trial deployment. In this paper, we report on results and insights gleaned from the development of a testbed infrastructure and phased experiments; (1) the design of a re-usable measurement-inference architecture into which 3rd party sensor developers can integrate a wide variety of ldquoanomaly detectionrdquo algorithms to derive the same correlation-related performance benefits; (2) the development of a series of validation methodologies necessitated by the lack of mature tools and approaches to attest to the security of distributed networked systems; (3) the critical role of learning and adaptation algorithms to calibrate a fully-distributed architecture of varied devices in varied contexts, and (4) the utility of large-scale data collections to assess what's normal behavior for Enterprise end-host background traffic as well as malware command-and-control protocols. Finally, we propose collaborative defense as a blueprint for emergent collaborative systems and its measurement-everywhere approach as the adaptive underpinnings needed for pervasive services.