Optimal frame structure design using landmarks for interactive light field streaming

Light field is a large set of spatially correlated images of the same static scene captured using a 2D array of closely spaced cameras. Interactive light field streaming is the application where a client continuously requests successive light field images along a view trajectory of his choosing, and in response the server transmits appropriate data for the client to correctly reconstruct desired images. The technical challenge is how to encode captured light field images into a reasonably sized frame structure a priori (without knowing eventual clients' view trajectories), so that at stream time, expected server transmission rate can be minimized, while satisfying client's view-switch requests. In this paper, using I-frames, redundant P-frames and distributed source coding (DSC) frames as building blocks, we design coding structures to optimally trade off storage size of the frame structure with expected server transmission rate. The key novelty is to facilitate the use of “landmarks” in the structure-popular reference frames cached in the decoder buffer-so that the probability of having at least one useful predictor frame available in the buffer for disparity compensation is greatly increased. We first derive recursive equations to find the optimal caching strategy for a given coding structure. We then formulate the structure design problem as a Lagrangian minimization, and propose fast heuristics to find near-optimal solutions. Experimental results show that the expected server streaming rate can be reduced by up to 93.6% compared to an I-frame-only structure, at twice the storage required.