A hybrid methodology combining strength reduction of Finite Element Method(FEM) with limit equilibrium theory is presented to evaluate rock slope stability by considering the demand for slope stability analysis and slope angle optimization of an open-pit, and it is applied to a case study for optimizing slope angle. The methodology uses the strength reduction of FEM to identify the plastic area in slope preliminary, which is supposed to be the most likely sliding surface. Thus the range of sliding surface inclination could be regularized. And then by taking this sliding surface as a start point of analysis, the limit equilibrium theory is used to calculate safety factors under different slope angles. By plotting the curve of safety factors with respect to different slope angles, the optimal inclination of a slope which is met to safety threshold could be determined. Finally, the upper slope located in the central part of open-pit iron mine, Cape of Lambert is presented to demonstrate this methodology. Results show that this approach is effective to deal with the analysis of rock slope stability.