Robust Smartphone Screen Integration with Deep Learning for Virtual Reality Pass-through

Virtual reality is revolutionizing immersive experiences, yet the seamless integration of everyday devices, such as smartphones, remains a challenging frontier. This paper presents a method for dynamically integrating a live smartphone screen into the virtual reality pass-through view. Our approach leverages a lightweight convolutional neural network (CNN) to detect the smartphone in real time, accurately determining its position, scale, and orientation within the camera feed. By synchronizing live screen capture with inertial sensor data, our system computes precise affine transformations that ensure the overlay remains perfectly aligned with its physical counterpart, even during rapid movements. Experimental evaluations demonstrate that our solution achieves an average of 29.2 detection runs per second, delivering a stable and high-fidelity integration without the need for additional hardware. Evaluations also showed a 0.44 increase in detection F1 score during live comparisons to the baseline alternative, which does not use deep learning. This dynamic overlay enhances visual clarity and interaction in virtual reality and bridges the gap between virtual and real-world interfaces, empowering users to access notifications, messages, and productivity applications seamlessly.