Application of Big Data File Segmentation, Uploading, and Breakpoint Continuation in Monitoring the Real-Time Data Transmission Status of Ocean Observation

Marine observation data are an important basis for conducting marine scientific research and environmental monitoring. However, in practical application, due to the huge amount of data, as well as in real-time transmission is often subject to the network environment, equipment and other factors, resulting in the interruption and packet loss of data in the process of transmission, so how to realize efficient and stable data transmission is an urgent problem to be solved. In order to address this problem, this paper proposes a real-time delivery method for big data based on the combination of segmented uploading and breakpoint continuous transmission. Firstly, this paper divides the ocean observation data and dynamically adjusts the size of each division according to the network conditions and transmission requirements; secondly, this paper adopts the break-point continuous transmission technology, so that when there is an interruption in the transmission process, the transmission can be continued from the break-point in order to ensure that the data are complete and continuous; finally, this article establishes a real-time data transmission status monitoring system, which realizes real-time monitoring of the entire transmission process and timely detection and processing of abnormal conditions during transmission. The speed of the system in this paper fluctuates between 50 and 100 MB/s, while the speed of the traditional system mainly hovers between 20 and 50 MB/s. The research of this paper can effectively improve the transmission efficiency and stability of ocean observation data, reduce the data loss due to network fluctuations, equipment failure and other reasons, and lay the theoretical and technical foundation for the development of marine scientific research and environmental monitoring and other fields.