This paper presents a new approach to using locally linear embedding (LLE) method in object tracking problems. By means of measuring the divergence of the K nearest neighbors of test data, a novel method is proposed to distinguish object from background directly through the LLE embedding results. Avoiding training a mapping function, this approach is less dependent on a beforehand training set of object compare to other attempts of utilizing manifold embedding method on object tracking. Besides, an asymmetric version of LLE is derived to improve the tracking performance. A Bayesian inference framework is built to apply this approach to visual tracking problem using particle filter. Experimental results demonstrate both efficiency and adaptability of our algorithm.