It is difficult to extract the characteristics of matching points because the image intensity in multi-source images is non-linear with respect to pixels. A multi-source remote sensing image feature point extraction algorithm based on the principal oriented component analysis is proposed. In the proposed algorithm, a principal analysis is first made for the multi-source remote sensing images, Secondly, a scale space is built with the first principal components of the multi-source remote sensing images; and the scale parameter is increased to improve the matching accuracy as well as the operation speed of the matching of feature points. This algorithm can extract a certain number of the correct matching point pairs. Comparative analysis of experiments shows that the proposed algorithm enjoys higher precision and better stability than the general algorithms.