Abstract：Outdoor surveillance systems would be subjected to illumination changes and camera shakes to cause changes in the background. This paper proposes a background model and a method of removing camera shake effects based on edge features. The edge features of the background are handy to have a representation of the scene background invariant to illumination changes. Firstly, reliable background edges are extracted by edge detection algorithms from a video sequence. Secondly, the Gaussian Mixture Model (GMM) is employed in the regions near the background edges, while the Temporal Average Model (TAM) is used for other regions. Both the detection accuracy and speed are considered. Finally, to eliminate the false target caused by camera shakes, a method using the reliable background edges is proposed. A background update mechanism is also used. The experiment results show that the detection speed is increased by 50% than the GMM method. The detection accuracy is better than the TAM method. The advantages of the GMM and TAM methods are well combined in the new method. When the camera is shaking, the false alarm rate is reduced by using background edge features. Good detection results are obtained as compared to the GMM and TAM methods. The method can be used for moving object detection in complex scenes.