Real Time Transmission Monitoring and Alarm Mechanism of Big Data Ocean Observation Files Combined with Apache Kafka

This study explores the real-time transmission, monitoring and alarming mechanism of big data ocean observation files using Apache Kafka. The research aims to solve the problems of data loss, delay and system failure that may occur during the transmission of ocean observation data. By establishing an efficient data transmission channel and real-time monitoring system, and combining it with an intelligent alarm mechanism, the integrity and real-time nature of the observation data are ensured. Experimental results show that Kafka performs well in terms of data transfer rate, stability and system latency. The data transfer efficiency reaches 980 entries/second on average, and the data packet loss rate is less than 0.1%. In terms of data transfer stability, Kafka stays within ±2% fluctuation. The system latency is 50 ms on average, and the fluctuation range is within ±10 ms. From the conclusions of the experimental data, we can see the superior performance of the monitoring system based on Apache Kafka technology in high-frequency data transmission and real-time processing.