The combinatorial optimization occurs in many real-world problems including the fields of engineering, physics and economics. It has been recognized that some problems with highly degenerate states are difficult to solve in terms of many existing optimization algorithms. This paper proposes a novel stochastic method with modified extremal optimization (EO) and nearest neighbor search to deal with these problems. It starts from making use of the recent discovered statistical property to generate the initial configurations by the nearest neighbor search and then further explores the complex configuration spaces by a modified EO approach that applies more general probability distributions-based evolution mechanism. The experimental results with some hard instances of traveling salesman problem (TSP), a popular benchmark for combinatorial optimization problems demonstrate that the proposed method may provide better performance than other physics-inspired algorithms such as simulated annealing, EO and self-organized algorithm.