Research on Digital Twin Construction and Safety Management Application of Inland Waterway Based on 3D Video Fusion

The digital twin can reconstruct the scenes of real traffic operation of inland waterways in virtual space, giving birth to new ideas for the digital transformation and upgrading of water safety management. Aiming at the problems of video fragmentation, sub-system data separation and untimely emergency response in the current inland waterway safety monitoring, this paper proposes a new method for reconstructing and applying a digital twin of inland waterways based on 3D video fusion which consists of four parts: (1) Combining drone tilt photography and BIM modeling technology to construct a three-dimensional scene model of inland waterways; (2) Proposing a multi-channel video three-dimensional registration and rendering method to integrate in real time discrete surveillance videos of different angles of view with three-dimensional scene model to establish the spatial correlation between different video images in the scene; (3) Integrating multi-source IoT sensing data to form a scene of digital twin of inland waterways; (4) Exploring innovative application methods for water safety management based on digital twins to meet the water safety management requirements and business logic of inland waterways. A certain segment of the Changhu Shen Line was selected for demonstration and application. The application results show that the construction and application of digital twins for inland waterways combined with 3D video reduces the pressure on managers to understand real-time video footage and improves the efficiency of daily monitoring, evidence collection, and emergency response.


I. INTRODUCTION
With its comparative advantages such as large transportation capacity, less land occupation, small energy consumption, light pollution, and low cost inland water transport provides a reliable guarantee for the rapid economic and social development of coastal regions [1]. The safety of water transportation bears on the development of the national economy and social stability. The prosperity and development of inland water transportation put forward higher requirements for water safety management.
Inland waterways have typical characteristics such as narrow passages, excessive amount of bridges, shallow water depth, poor ship maneuverability, miscellaneousness of ships, The associate editor coordinating the review of this manuscript and approving it for publication was Razi Iqbal . and small transportation enterprises, all of which makes waterway safety supervision one of the most difficult issues in the field of waterway management. The use of digital and intelligent means to improve the efficiency and passing capacity of inland water transport and to ensure the safety of inland water transport is not only an important foundation for promoting the sustainable development of inland waterways, but also the new direction of and new requirements for the development of inland water transport.
The international water safety intelligent applications are mainly aimed at the offshore and ocean transportation markets, while the research on the application of digital safety supervision on inland waterways is far too scarce. Most of China's inland waterway safety monitoring uses high-point surveillance cameras, smart bayonet supervision systems, etc., which can achieve the supervision of key areas such as ship locks, bridge areas, and water service areas and have achieved good application results, but there are still some urgent problems to be solved: (1) video fragmentation: The deployment of surveillance cameras in the channel and along the coast is relatively scattered, the correspondence between videos and the waterway segment is difficult to remember and it is not easy to switch, track, and it is impossible to find problems in time and understand the overall situation; (2) subsystem data separation: video surveillance is separated from the business data of ship AIS, bayonet smart supervision and other subsystems, and the linkage is poor; (3) emergency response is not timely: manual video viewing workload is large, the cognitive pressure on staff from switching between scenes is high, and automatic video inspection and responding means are lacking.
The digital twin reconstructs the real traffic operation scene of the inland waterway in the virtual space [3], [4], which brings new ideas to the solution of the above problems. The thesis focuses on 3D video fusion-based inland waterway digital twin construction and safety management application methods. Specifically (1) building a static scene model of the waterway by making use of drone tilt photography and BIM technology; (2) integrating discrete surveillance videos with different viewing angles and three-dimensional models of the surveillance scene to form a spatial correlation between different video images in the scene; (3) integrating water safety surveillance multi-source IoT sensing data to construct digital twin scenarios for inland waterways. The development of digital twin-based water safety management applications can help reduce the pressure on managers to understand realtime videos and improve the efficiency of daily monitoring, evidence collection and emergency response.
Different from the traditional safety monitoring method of inland waterways, the main contributions of this paper include: 1) A method of 3D registration and real-time fusion of multi-channel video over water is proposed to realize integrated monitoring of virtual and real fusion in line with human cognition, and solve the problem of video fragmentation; 2) A three-dimensional aggregation and fusion method of water sensor data based on three-dimensional panorama is proposed, which realizes the real-time access and enhanced display of dynamic and static data of the Internet of Things, and avoids the problem of sub-system data separation; 3) A digital twin construction method for inland waterways based on 3D video fusion is proposed, which realizes 2D and 3D linkage water safety management based on digital twin scenes, and assists in improving the efficiency of emergency response.

II. RELATED WORKS
The use of new information technology to continuously improve water traffic safety and operational management efficiency has always been one of the important areas that water traffic researchers' focused on. This section analyzes the research and application status of water safety monitoring related technologies both in China and around the world.
There are many navigable rivers in the world, and the standardization of inland water transportation in the United States and Western European countries has developed to a high degree. The core inland navigation network in Europe is about 10,000 kilometers, which is composed of four water transportation channels including the Rhine River and its tributaries, the Danube River and its tributaries, the West Water Transportation Channel, and the South Water Transportation Channel. In 2006, the European Union fully implemented the European Inland Waterway Shipping Integrated Service System (RIS) [5] and formulated technical standards for transnational shipping data information exchange and ship identification and tracking, by which it accelerated the process of European inland waterway coordination. The Dutch ship reporting information system (IVS90) [6] supports ship lock dispatch, ship traffic management and water safety monitoring. The shipping information service system (DoRIS) [7] in Austria is used for shipping management, ship lock scheduling, ship navigation and water safety supervision. Hungary's shipping information service and emergency call service (DISR) [8] is used for maritime safety monitoring and rescue support. The inland water monitoring system invested and constructed by the Federal Government of Germany [9] has a complete database, which regularly reports the water level of the waterway and the working conditions along the river 4 times a day, and is responsible for dangerous goods transport ships, large ships over 140 meters, and towing vessels with limited maneuvering capacity will be tracked throughout the entire journey. The total length of the U.S. inland waterways is about 41,000 kilometers [10], which is mainly composed of five major water systems: Moby River, Columbia River, Mississippi, Gulf of Mexico, and Atlantic Coastal Waterways. The Mississippi River is the main artery of North-South shipping in the United States, the total mileage of main and branch routes are 25,900 kilometers. Since 2003, with the goal of building intelligent waterways, digital information management in the United States has gradually matured, and electronic databases, port and waterway security systems have been widely put to use. The United States has established the world's largest vessel traffic service system (VTS) [11], which is a component of the US Port and Waterway Security System (PAWSS). The US Coast Guard Research and Development Center has carried out research on the Intelligent Waterway System and Waterway Information Network (IWS/WIN) [12] and has developed and established a complete IWS system for water transportation safety and supervision.
China's inland waterways are mainly distributed in the Yangtze River, Huai River, Pearl River, Heilongjiang River and other water systems. As of the end of 2020, the navigable mileage of inland rivers is 127,700 kilometers, of which the Yangtze River system is 64,700 kilometers. In terms of maritime safety supervision, China has constructed maritime three-dimensional supervision system on a basic level [13], [14], which is composed of the automatic identification system (AIS), global satellite positioning VOLUME 9, 2021 system (GPS), vessel traffic management system (VTS), China's long-range ship identification and tracking system (LRIT) and closed-circuit surveillance television (CCTV) system [15]. In 2014, the Shanghai Maritime Safety Administration completed the video surveillance supplement and the renovation project in the waters under the jurisdiction of the Huangpu River [16]. It adopted brand-new starlightlevel high-definition network surveillance equipment and installed a surveillance camera every 3 kilometers in the entire water area to improve the clarity of surveillance video and the visibility of night surveillance, and developed a security command three-dimensional simulation subsystem, which provides high-quality methods for daily supervision. In September 2019, the Yangtze River Digital Channel Dynamic Monitoring Platform was fully integrated and operated [17], which realized the interconnection of dynamic monitoring information such as navigation marks, water regimes, control sections, and channel dimensions of the 2687.8-kilometer main channel of the Yangtze River and the digital management transformation into ''See from afar, Manage While Seated, Use on the Go''. Jiangsu Province has built the ''Perceptive Channel'' information project in the Wuxi section of the Sunan Canal [18], and set up 25 fixed monitoring facilities on both sides of the 40-kilometer channel. Each monitoring point uses 2-3 high-definition digital cameras with not less than 20 to 36 times of optical zooms, which is responsible for fixed and omni-directional tracking and the video recording of the upstream and downstream segments, providing full-region, full-process, and full-scale timely, dynamic, and accurate monitoring, management and services for the passage of the channel. China Transportation and Communication Information Center [19] made use of the characteristics of AR technology and interactively integrated the AR panoramic dynamic supervision platform and the video management platform, and consolidated the data information in the scene video management platform and the static targets and associated descriptions in the scene by means of virtual tags, which greatly improves the management efficiency. Hikvision has developed a smart maritime cloud image three-dimensional prevention and control system [20] to obtain panoramic video of monitoring points and link it with low-point cameras in the viewing field to achieve large-scale three-dimensional monitoring and video linkage that focuses on both the whole picture of the waterway and the specific segments. Picture-in-picture displays low-point camera video, which is query-able, searchable, positionable, describable, alarmable, and linkable and improves the application mode of the monitoring system and the efficiency of actual work.
The development direction of water safety monitoring of inland waterways will inevitably move towards highdefinition, intelligence, and integration [21]- [23] which means away from the traditional passive recording and post-verification video surveillance system to the intelligent type of safety monitoring system development that consists of pre-warning, automatic tracking, and intelligent analysis.

III. REVIEW OF METHOD
Based on the analysis of the characteristics of water traffic scenes and safety management requirements, this section elaborates the research ideas of the paper. The scenes involved in water safety monitoring mainly include: important flight sections, water service areas, hazardous chemicals docks, anchorages, traffic maintenance areas, bridge areas, smart bayonet gates, ship locks, ferry ports, wharves, water intakes, navigable bridges, etc. The main features of this type of scene include: open water surface monitoring area, low overall coverage; complex water environment, limited camera deployment conditions; complex layout of key areas, usually requiring a mixed deployment of box cameras, dome cameras, wide-angle, ultra-wide-angle, panoramic cameras, etc. there are few landmarks that can be referenced on the water, and the similarity of the monitoring pictures is very high; the video is fragmented, and the direct recognition pressure is high. The managing unit of the inland waterway is the integrated traffic and water law enforcement team and the port and shipping management agency. The key objects of water safety supervision are ships, inland waterways and navigation infrastructure. The purpose of supervision is to ensure the normal passage of ships, the smooth flow of waterways and the safety of operation of navigation. Therefore, the business requirements for water safety management mainly involve four aspects: facility management, channel protection, illegal evidence collection, and application response.
Combining with the characteristics of water monitoring scenes and safety management requirements, in view of the current problems of video fragmentation, subsystem data sharing and untimely emergency response in the current inland water safety monitoring, this paper proposes a digital twin construction and safety management method of inland waterways based on 3D video fusion. The main steps are shown in Figure 1.
As shown in Figure 1, (1) UAV tilt photography and BIM modeling are used to realize the three-dimensional reconstruction of multi-scale water surveillance scenes, forming the basis of the static model of the digital twin; (2) Proposing real-time fusion method of three-dimensional registration and registration of multi-channel surveillance video, and integrate video images in a three-dimensional virtual scene in a texture manner to form a digital twin scene; (3) Consolidating multi-source dynamic and static IoT sensing data of water security monitoring scenes and display them in the form of VR labels and update the corresponding status of the digital twin scene; (4) using the constructed digital twin scene to carry out safety management applications such as inland waterway facility management, daily monitoring, illegal evidence collection, and emergency response.

IV. CONSTRUCTION OF A WATER SCENE MODEL BASED ON BIM AND DRONE TILT PHOTOGRAPHY
The construction of the 3D model of the scene is the basis for the construction of the digital twin scene of the inland waterway. The water monitoring scene includes many types of structural objects such as waterways, navigation buildings, water surfaces, and surrounding buildings. Different structural objects require different modeling methods. The channel and both sides of the channel have a large scene range and high realism requirements. UAV tilt photography is used to achieve rapid scene reconstruction. The water surface of the channel is opening, but it is not the focus of safety management. The water surface is constructed directly using high-definition satellite images as textures. Ship locks, bridges, wharves and important navigation buildings are the focus of asset supervision whose data accuracy requirements are high which will be collected through BIM modeling. Multiple modeling methods are combined to achieve multi-scale 3D reconstruction of large-scale water monitoring scenes.

A. USING DRONE TILT PHOTOGRAPHY FOR CHANNEL RECONSTRUCTION
Using drone tilt photography technology to collect aerial three-dimensional data and to build a 3D model of the scene within a certain range of inland waterways and surrounding areas. Specifically, by making use of aerial triangulation, the image import, dense point cloud generation, grid construction, patch drawing, and automatic texture mapping, model singulation, model modification, 3D model output, etc., a large-scale water scene basic model can be constructed, as shown in Figure 2, using image rapid modeling tools for refined monomer processing and Mesh model modification to further improve the accuracy and quality of 3D modeling.

B. MODELING OF NAVIGATION BUILDINGS BASED ON BIM
For key elements such as docks, ship locks, and water service areas, BIM is used to build models with higher accuracy to compensate for the incomplete acquisition of UAV tilt photography data, low model accuracy and loss of structural details. Using professional modeling software such as VOLUME 9, 2021 Revit, Infraworks, Civil 3D, etc., BIM modeling is carried out on structural objects such as ship locks, bridges, and water service areas, as shown in Figure 3. Replace the model reconstructed by drone tilt photography with BIM model replacement, as shown in Figure 4. It can be seen that the topological relationship of the architectural structure of the water service area after the replacement is clearer, and the accuracy of the 3D model of the scene is improved.

V. THREE-DIMENSIONAL REGISTRATION AND REAL-TIME FUSION OF MULTI-CHANNEL SURVEILLANCE VIDEO FOR WATER TRAFFIC SCENES
In order to establish the temporal and spatial correlation between surveillance videos, a three-dimensional registration and real-time fusion method of multi-channel surveillance videos of water traffic scenes is proposed. The image is not deformed, but is calculated through a single photo of the scene and the parameters of the camera to establish the projection relationship between the video image and the model texture, which will be projected on the 3D video model in the scene in real time. The main steps include: (1) Converting the position and posture information of the camera in the real environment into position values and posture values in threedimensional space; (2) Using the converted position and posture value to calculate the camera's model-view matrix in three-dimensional space and projection matrix; (3) Using model-view matrix and projection matrix to calculate the view cone structure of the camera in three-dimensional space; (4) GPU parallel accelerates the fusion process; (5) Using model -view matrix and projection matrix to render the depth information of the scene under the camera's point of view before fusing it again; (6) Fragment texturing and coloring operations are performed in the graphics card, and the fragments undergo raster operations before it is finally converted to the pixels seen on the screen. Examples of the rendering effects of camera 3D registration and fusion in different scenes are shown in Figure 5. Figure 5(a) shows the camera annotation and video fusion effect of the surveillance video on the top of a ship lock, and Figure 5(b) shows the fusion effect of the coastal surveillance video of a water service area. The fusion scene allows users to observe the fusion result from any virtual point of view to achieve more free interaction and application. This method enhances the authenticity of the virtual scene, reduces the pressure of detailed modeling and scene dynamic simulation, and visually displays the state and response of real objects.

VI. CONVERGENCE FUSION AND ENHANCED DISPLAY OF MULTISOURCE IOT SENSOR DATA
The water safety monitoring scene contains multi-source sensing and business management data of four types of elements: people, ships, waterways, and the environment, with high integration costs and complex real-time linkage According to the daily monitoring business needs of water safety, access to facility attributes, location labeling, ship traffic information, weather, ship AIS data, RFID, temperature, humidity and other sensor data are established. And these relatively scattered spatial sensor data will be displayed in the 3D virtual and real fusion screen, helping managers to more accurately control the scene situation. Managers can also independently set Point of Information (POI) descriptions, increase hot spots, hot chains, and real-time data of various sensors and a variety of chart displays and multi-source data, which can be seen in the virtual reality world. The actual display example effect is shown in Figure 6. Figure 6(a) shows an example of labeling facilities and equipment such as surveillance cameras, smart single-phase charging piles, smart water intake facilities, sewage treatment equipment, and water gas stations in a water service area scene, and can dynamically monitor the status of facilities and equipment and reflect this information in the digital twin scene in real time, and an alarm event can be formed for abnormal data according to business requirements, and timely response; Figure 6(b) shows the AIS dynamic data augmented reality annotation of the ship passing the gate as sensed by the smart ship lock monitoring system As a result, the smart ship lock recognizes ship information through AIS, RFID, image recognition and other means, and aggregates the ship's basic information, cargo information, and violation information in real time into the data twin scene and presents the data in the form of augmented reality.

VII. WATER SAFETY MANAGEMENT APPLICATION BASED ON DIGITAL TWIN
Utilizing the constructed digital twin scenario avoids the problems of video fragmentation and subsystem separation in traditional security surveillance, and provides a data and model basis for timely response to emergency incidents. The application of new technologies will inevitably bring about changes in the corresponding management model. Therefore, it is necessary to further innovate the management model and improve the management efficiency. Water safety management applications based on digital twins are mainly embodied in the following aspects.
Three-dimensional digital asset management: Connect 3D integration and asset labeling of important IoT sensor equipment and facilities such as ship locks, anchorages, dangerous cargo wharves, water service areas, bridges, and maritime checkpoints and the shore power, smart water supply, and sewage discharge in the water monitoring scene, smart bayonet, etc. are connected to the three-dimensional scene in real time in the form of dynamic tags to realize the three-dimensional digital asset management of important water transportation facilities and equipment. In the digital twin scene, the status information of facilities and equipment can be viewed intuitively, combined with data analysis, the status of the facilities and equipment can be judged, and faults can be detected and dealt with in time.
Channel intelligent automatic cruise: set up scenes such as important flight sections, easily congested points, water service areas, anchorages, dock berthing stalls, dangerous areas, traffic maintenance areas, bridge areas, ship locks, accidentprone areas, smart bayonet points, etc. Fixed cruise points and cruise lines, in a mode that conforms to human cognition and habits, realizing automatic cruises in key areas on the water VOLUME 9, 2021 in three-dimensional space, reduces the frequency of routine cruises of sea cruisers, improves the cruise targeting and cruise efficiency, and improves the management and control of navigable ships in the waters of the jurisdiction ability.
Violation evidence collection supported by threedimensional panoramic view: daily monitoring of water safety operations in three-dimensional scenes, combined with AIS, VTS and other system data and alarms, to quickly correlate the alarm information and the location of the illegal ship in the three-dimensional space, making it more intuitive and convenient in early warning, discovery, evidence collection and handling of violations.
Two-dimensional and three-dimensional linkage emergency response: when there are abnormal conditions such as water traffic jams, accidents, anchoring, rescue alarms, etc., according to the location of the alarm or abnormal situation, the two-dimensional and three-dimensional linkages can be quickly dispatched in the three-dimensional space to check and confirm the related videos and support the viewing of the upstream and downstream status of channel congestion points, discover and analyze the causes of congestion, and realize more intuitive and convenient abnormal event discovery, response and handling.
Off-site law enforcement based on three-dimensional panoramic view: combined with the smart bayonet system, the two-dimensional AIS system and three-dimensional space coordinates are uniformly calibrated to realize the linkage of two-dimensional and three-dimensional maps, videos and scenes, according to dynamic information such as the location of the illegal ship. It can be quickly linked to the corresponding three-dimensional space for viewing and confirmation, supporting the violation confirmation, evidence collection and off-site law enforcement.
Digital twin scenes can also be used for planning rehearsals for full coverage of channel video, dynamic identification of berths in service areas, and verification of sneak shots of ships. In addition, in order to meet the purpose of all-weather digital twin monitoring of key channel section, important nodes or key ships (such as dangerous cargo terminals), infrared cameras, thermal imaging or dual-spectral cameras need to be deployed to enhance night monitoring capabilities.

VIII. DEMONSTRATION AND EFFECT ANALYSIS
A channel of about 2 kilometers upstream and downstream of a ship lock on the ChanghuShen line was chosen to carry out the demonstration and the effect analysis of the digital twin construction and safety management of the inland waterway.

A. SELECTION AND IMPLEMENTATION OF DEMONSTRATION APPLICATIONS
The demonstration scene is rich in typical structural objects, as shown in Figure 7, including 1 ship lock, 2 navigable 4-fork ports, 1 5-fork port, 1 water service area, 1 management checkpoint, 1 maritime dedicated dock, and 1 smart ship lock system and 3 cross-channel bridges. The demonstration channel has various camera types and good hardware conditions. There are 25 surveillance videos, including 20 bullet cameras, 3 dome cameras, 1 four-lens panoramic camera, and 1 wide-angle camera. Demonstration scene IoT facilities and equipment in the demonstration are diverse, including 1 smart bayonet system, 14 shore power piles, 12 water replenishment facilities, 4 sewage discharge facilities, 1 sewage treatment facility, 1 smart trash can, and 1 gas station that is under construction and 1 smart express cabinet in China, the selected scene is representative of the real world. Selection has good representativeness. Figure 8 shows the digital twin construction process of the demonstration waterway scene. Figure 8(a) shows the 3D scene reconstructed by drone tilt photography for the demonstration waterway scene. Using the Qianxun Q10 flying platform, the aerial resolution is 3cm, the flight takes 1 sortie, and it takes 48 minutes to collect more than 10,000 tilted photographic images to construct a static basic model of the digital twin scene; Figure 8  scene. The CAD drawing of the navigation building and the software such as Revit is used to carry out the more accurate BIM modeling of the important navigation building and slight processing of the original model; Figure 8(c) shows the effect of using BIM models to replace oblique photography reconstruction models, fusing oblique photography, highdefinition remote sensing satellite images, BIM and other data and models to achieve a large range construction of the water scene; Figure 8(d) shows the effect of multi-channel video 3D registration and fusion drawing, information such as the position, height, orientation, type and other information of 25 surveillance cameras are collected to establish a threedimensional scene and two-dimensional surveillance video data. The original video is projected onto the 3D video model in the scene in real time by using the association relationship; Figure 8(e) is a collection of multi-source IoT sensing instal-lations such as facilities, equipment, navigation elements, and smart bayonet. Through the convergence and fusion of multi-source data, real-time annotation and augmented reality display of the dynamic and static data of the IoT sensor in the three-dimensional space is achieved. By the implementation of the above steps, the scene model, multi-channel video, and object transmission are integrated in real time and the digital twin scene of the inland waterway monitoring area is constructed using multi-source data such as sensory sensor, as shown in Figure 8(f).

B. DEMONSTRATION APPLICATION EFFECT AND EVALUATION
Based on the constructed digital twin scenarios and business requirements, a water safety monitoring system based on digital twins is developed. The system interface is shown  in Figure 9. The digital asset management, daily monitoring of water safety, illegal evidence collection, and emergency response effects of twin water safety applications are analyzed.
Facilitating the overall monitoring of smooth waterway and reducing the cognitive pressure of monitoring personnel. As shown in Figure 10, the digital twin scene establishes the spatial association relationship between the 2D video and the 3D scene, providing a virtual top view of the global traffic status monitoring of important areas, which makes it possible for users to be able to intuitively view the running status of water traffic and supports autonomous roaming view analysis of the 3D space, reducing the cognitive pressure of the monitoring personnel.
Smart video cruising across the who waterline, reducing the cost of manual cruising. Figure 11 shows the automatic video cruise effect of the channel. Using the constructed digital twin scene of water traffic, the user can customize the cruise line, cruise point, cruise frequency, and cruise time in the scene. The system will follow the set smart cruise line to carry out a full-line automatic video cruise in line with ship navigation awareness, which reduces the number of manual on-site cruises and manual switching views and the cost of manual cruises.
The panoramic view of historical video can be traced back to improve the efficiency of illegal evidence collection. Figure 12 shows that the water security surveillance system integrated with real-scene video controls all surveillance videos in the scene with a unified clock, and can go back in time   according to user needs. Time is used to trace or obtain evidence for the entire process of occurrence, development and handling of an abnormal event, which improves the efficiency of traceability and tracking of ship violations.
Convergence and integration of data on water facilities and equipment to realize three-dimensional digital asset management. Figure 13 shows an example of digital asset management in the water service area of the demonstration channel. The 3D panoramic video fusion water safety monitoring system supports navigation building information, multi-source IoT sensing equipment, and real-time access of status and data to achieve water three-dimensional digital asset management of general navigation buildings, important electromechanical facilities, and special equipment in the monitoring scene.
The two-dimensional and three-dimensional linkage response of maritime safety alarms improves the ability of accident rescue and emergency response. Figure 14 shows the two-dimensional and three-dimensional linkage response  effects. The system includes two-dimensional maps and three-dimensional scenes. When an abnormality is detected, it can quickly carry out moves such as skipping, confirming roaming, and viewing between the two-dimensional small map, three-dimensional scene, and split-screen video, so as to support the ability of the water safety management unit to confirm the police situation, abnormality location, accident rescue and emergency response.
Further comparison of the monitoring effects before and after the demonstration of the demonstration channel. Figure 15(a) shows a two-dimensional matrix water safety monitoring screen. It can be seen that the video on the large monitoring screen is fragmented, and it is difficult to effectively find the position of each video in the real world. Figure 15(b) shows the three-dimensional surveillance picture after panoramic video fusion. Because the position of each video in the three-dimensional space is established, managers can directly correspond to the position of each video in the real world moreover, without having to consider the specific point of the video, managers can use the three-dimensional space as the basic scene of daily management, and automatically schedule the video scene of the target area of interest for daily management and auxiliary decision-making. Figure 16 shows the comparative effect of manual video switching and smart video cruise. Figure 16(a) shows the scene of manual video switching for different scene viewing and round patrol. It can be seen that most of the scenes are bolt and dome cameras. The switch between different types of cameras can be realized through the combination of keyboard, mouse and joystick; Figure 16(b) shows the scene of intelligent maritime automatic video cruise based on three-dimensional panoramic view. By presetting patrol paths and patrol points, automatic video cruise is realized in the three-dimensional scene of the entire jurisdiction at a fixed time or at a certain frequency, and conduct a safe cruise of the surveillance scene in a way that is more in line with the cognition of managers. Figure 17(a) shows the AIS system interface of water security monitoring. It can be seen that the video monitoring and business management platform are separated from each other. Only the relationship between the camera points is VOLUME 9, 2021  established in the two-dimensional channel map, and click the camera label can view the corresponding video. Figure 17(b) shows an example of integrated management of video and AIS data. The AIS status and data of ships are connected to the three-dimensional panoramic space to achieve the convergence and integration of AIS data, enhanced display and decision-making assistance analysis. Figure 18(a) is the monitoring picture of a wide-angle panoramic camera at a checkpoint. It can be seen that the picture has obvious distortion. Figure 18(b) is the effect of 3D panoramic fusion. It can be seen that the distortion is not obvious after correction. The processed image is more in line with people's visual observation habits.
In addition, the digital twin-based water safety monitoring system also supports functions such as multi-level control, space tracking, camera relay, magnifying glass, etc., to implement services in a three-dimensional space that is more in line with the perception of managers.

IX. CONCLUSION
Aiming at the problems of video fragmentation, sub-system data separation and untimely emergency response in the safety monitoring of inland waterways, based on the analysis of the characteristics of water traffic scenes and the needs of safety management applications, a digital twin construction and application method based on 3D video of fusion inland waterway is proposed. UAV tilt photography and BIM technology are used to build an inland waterway water scene model to form the basis of a multi-scale static model of the scene; a three-dimensional registration and real-time fusion method for multi-channel surveillance videos is proposed, which combines three-dimensional discrete surveillance videos with different viewing angles to form the spatial correlation between different video images in the scene; the multi-source dynamic and static IoT sensor data of the water safety monitoring scene is gathered and integrated to construct the digital twin scene of the inland waterway to carry out water safety management applications based on digital twins, such as facility management, daily monitoring, illegal evidence collection, and emergency response. This paper also takes the upstream and downstream segments of a ship lock on the Changhu-Shen Line as an example, and carries out demonstration applications. The results show that: through three-dimensional panoramic virtual and real fusion display, two-dimensional \ three-dimensional two-way linkage, video data synchronization control, smart maritime automatic cruise and other technical applications and system research and development, the problem of fragmentation and lack of relevance of traditional sub-camera monitoring picture can be solved, and the overall change situation mastery of a large continuous area can be achieved; automated video inspections of important flight segments have reduced the workload of manual on-site inspections and manual switching inspections; the three-dimensional integration of water multisource sensing data has been realized, and the rapid detection of emergency incidents and the two-and three-dimensional linkage response capabilities have been improved. The next step is to focus on the digital twin scene iteration driven by online data, the simulation and fusion drawing of water traffic operation, and the update of the water environment driven by monitoring data.  Since 2017, he was a Research Assistant with China Academy of Transportation Sciences, Ministry of Transportation China. His research interests include microscope traffic simulation, driver behavior analysis, person following control, and building information modeling (BIM).
LIN ZHU was born in Rongcheng, Xiong'an, China, in 1984. She received the B.S. degree in computer science and technology engineering from Southwest University, in 2008. She qualified as a Traffic Information Engineer in 2013 and a Software Testing Engineer in 2014. Since 2008, she has been engaged in traffic informatization work. Her research interests include software platform development and testing, BIM scene rendering, data processing, road maintenance consulting, and technical transportation science. She qualified more than ten relevant awards and more than ten inventions.
ZIYI LV was born in Xinji, Hebei, China, in 1994. She received the M.S. degree in engineering from China University of Mining and Technology, Beijing, in 2020. Since 2020, she has been serving as a Research Intern with China Academy of Transportation Sciences. Her research interests include transportation virtual reality and transportation data virtualization.