Due to the acoustic attenuation in the human body, an efficient gain compensation on the ultrasound image is necessary for a better imaging quality in a medical ultrasound imaging system. The traditional manual adjustment method suffers some drawbacks, such as the difficulties in adjusting a special region, so it is very important to implement the automatic time gain compensation (ATGC) algorithm in the clinical ultrasound imaging system. Because of the massive computation involved in this ATGC technique, this problem becomes the bottleneck for a clinical real-time imaging system. In this paper, a new parallel algorithm of ATGC based on Fermi GPU (graphics processing unit) is presented. The main procedures of this algorithm include the pre-processing, the speckle detection, the tissue intensity computation, the 2-D surface fitting and the adaptive gain compensation. The key parallel algorithm includes a parallel box filter with coarse-grained, parallel local variance coefficient computation, the optimized parallel matrix transposition, the parallel matrix inversion based on the LU factorization in a coarse-grained parallel way. Test results not only show that the output of the graphics processing unit (GPU) is definitely the same as that of the CPU, but also demonstrate an obvious speedup by using the GPU, that is, with 427 frames per seconol for the image size (512×261) , 267 times faster than the CPU implementation.