Determining Wave Height from Nearshore Videos Based on Multi-level Spatiotemporal Feature Fusion

Wave height is one of the most significant parameters of waves, affecting marine navigation, coastal environment, and surfing activities. In this paper, we present a wave height determination network from nearshore surveillance videos. This method consists of three parts. Firstly, video frames and frame differences are extracted to represent static and dynamic characteristics of waves, respectively. Then, by taking the two types of data as different modalities, two independent Network In Network (NIN) networks are constructed to learn spatial and temporal features of wave heights. Meanwhile, the two types of features are fused across multiple feature levels using a central network, where concatenation or attention-based fusion is applied. Finally, the wave height is determined by minimizing a global loss. The proposed network for wave height determination, dubbed NINMf-WH or NINMf-A-WH (with attention), was evaluated by a set of comparative experiments from two aspects: i) the basic network of feature learning: NIN vs. CNN; ii) the spatial-temporal feature fusion strategy: multi-level fusion vs. late fusion vs. intermediate feature fusion. The results show that NIN is suited to learn wave height features due to the nonlinearity of waves in video, and the multi-level spatiotemporal feature fusion can achieve low error and good stability of wave height determination. With attention-based fusion, NINMf-A-WH can further reduce the prediction error. To verify the generalization performance of the proposed network, we applied it for estimating wave heights in new surveillant videos. The results provide a low relative error of 6.43%±4.9, which can well meet the operational requirement for nearshore wave forecasting (<=20%).