Audio surveillance for self-aware environments

We present a state-of the art system that integrates diverse cutting-edge technologies to achieve real-time, distributed audio surveillance. The system employs a network of microphones mounted on ESP32 platforms, which transmit compressed audio chunks via MQTT protocols to Raspberry Pi devices for acoustic classification. These devices host an AST audio transformer model trained on the AudioSet dataset, enabling the real-time classification and timestamping of audio events with high accuracy. The output in JSON format is parsed into graph structures that encapsulate the annotated soundscape, providing a rich and dynamic representation of audio environments. These graphs are subsequently traversed and analyzed using Large Language Models (LLMs) and dedicated Python code, enabling the system to answer complex queries about the nature, relationships, and context of detected audio events. This approach combines the robustness of distributed sensing, the precision of transformer-based audio classification, and the interpretive power of LLMs, paving the way for advanced applications in acoustic surveillance, environmental monitoring, and beyond.