DDS: an efficient dynamic dimension selection algorithm for nearest neighbor search in high dimensions

The nearest neighbor search problem is defined as follows: given a set P of n points, answer queries for finding the closest point in P to the query point. In the past few years, there has been increasing interest in performing similarity search over high dimensional data, especially for multimedia applications. Unfortunately, most well-known techniques for solving this problem suffer from the "curse of dimensionality" that means the performance of the system scales poorly with increased dimensionality of the underlying data. The refined algorithms typically achieve a query time that is logarithmic in the quantity of points and exponential in the number of dimensions. However, once the number of dimension exceeds 15, searching in k-d trees or related structures involves the examination of a large fraction of the search space, thereby performing no better than exhaustive search. In view of this, we propose an efficient dynamic dimension selection algorithm to improve the performance of the nearest neighbor search especially in high dimensions.