Optimal Indexing: An Efficient Feature-Based Indexing Framework for Similarity Data Sharing at the Network Edge

Edge storage systems have drawn many efforts to extend the storage and service capabilities of cloud data centers. A pivotal aspect lies in the data-sharing mechanism, which integrates geographically dispersed weak edge servers into an efficient storage system. It enables users to launch data operations at any server and retrieve the desired data across the distributed system. However, it remains open to meeting the increasing demand for similarity retrieval across edge servers. The intrinsic reason is that the existing solutions can only return an exact data match for a query while more general edge applications require the data similar to a query input from any server. To fill this gap, this paper pioneers the similarity edge data sharing mechanism, a new paradigm to support high-dimensional similarity search at network edges. First, through deeply thinking about the nature of similarity data sharing, we propose the problem of Optimal Indexing and formulate it as the optimal transport problem from the data space to the network space. On this basis, we propose Prophet, the first known architecture for similarity data indexing at the edge. We first divide the feature space of data into plenty of subareas, then project both subareas and edge servers into a virtual space where the distance between any two points can reflect not only data similarity but also network latency. When any edge server submits a request for data insert, delete, or query, it computes the data feature and the virtual coordinate; and then iteratively forwards the request via greedy routing based on the forwarding tables and the virtual coordinates. By Prophet, similar high-dimensional features would be stored by a common server or several nearby servers. Compared with distributed hash tables in P2P networks, Prophet requires to visit logarithmic servers for a data request and reduces the network latency from the logarithmic to the constant level of the server number. Evaluation results indicate that Prophet achieves the comparable retrieval accuracy and significantly shortens the query latency compared with centralized schemes, while the load balancing performance is nearly optimal.