An Introspection Mechanism to Debug Distributed Systems

Distributed systems are hard to debug due to the difficulty to collect, organize and relate information about their behavior. When a failure is detected the task to infer the system's state and the operations that have some connection with the problem is often quite difficult and usual debugging techniques often do not apply and, when they do, they are not very effective. This work presents a mechanism based on event logs annotated with contextual information, allowing visualization tools to organize events according to the context of interest for the system operator. We applied this mechanism to a real system and its the effort and cost to detect and diagnose the cause of problems was dramatically reduced.