The background subtraction is an important method for detecting the moving objects, which is widely applied in the video monitor system. When background is occupied by a foreground for a long time, the foreground will be mistakenly regarded as a background. In order to solve the background reconstruction in which background does not always appears with the largest appearance frequency, a new background reconstruction algorithm based on neighboring correlation is proposed. At first, the data is sorted in an ascending or descending order; secondly, the sorted data is classified by the simple method; thirdly, the appearance frequency of classified classes is computed. The definite identity of background is obtained by appearance frequency. The candidate backgrounds are selected for the pixels without definite background; finally, background selection procedure based on neighboring correlation is repeatedly executed to the pixels until the background of all pixels has been obtained. Simulations results show that the algorithm is able to deal with the complex scene in which the background has been covered for a long time. The proposed algorithm is able to reconstruct the background of scene well, and therefore the target could be perfectly extracted and successfully tracked.