Real-Time Instance Segmentation Tracking Algorithm in Mixed Reality

In a mixed reality environment, in order to complete the interaction of manipulating virtual items with physical wooden sticks, a real-time and accurate object tracking algorithm is needed. Therefore, we design a fast, pixel-wised object tracking model to quickly and accurately segment wooden sticks in each frame. The model consists of two parts: The first part is an object detection model, which is responsible for identifying and detecting the bounding box of the object in the first frame; The second part is the instance segmentation model, which uses the bounding box of the object obtained in the previous frame and the current frame’s image features (extracted by the convolutional neural networks) to calculate the object boundary points in the current frame. In addition, we use dynamic convolutions in the CNNs to increase the representation capacity of the feature extraction parts without increasing the depth or width of the network. Experiments show that under the task of tracking stick objects in a mixed reality environment, our method achieves competitive performance in real-time running speed and segmentation quality.