An approach of optimal generation rescheduling for security operation of power systems is proposed with a heuristic constraint of transient stability by setting the maximum generator angle difference less than a threshold value in a designated post-fault duration. With this constraint, a concept called stability performance index is introduced in the approach for developing optimal control model. Rules for evaluating the stability performance index are presented. A technique of inverse integration is utilized to evaluate the rescheduling actions by using the gradient of the index. This approach has the advantage that it is applicable to systems represented by complex detailed models. Case studies on the 10-machine New England test power system are given to show the effectiveness and efficiency of the approach. The results suggest that this approach is able to ensure the system operation security in avoiding any potential transient instability.