Based on the ground pressure and strata control theory with "the strata movement and abutment pressure" as the core, the stope structural mechanical model and the gob-side retaining entry mechanical model are built, the caving zone height is determined, the non-compressed and compressed gangue deformation calculation formula is obtained, the compression effect of coal gangue in the mined-out area is explained and the gob-side retaining entry surrounding deformation is analyzed. For the coal gangue with three kinds of size ranks（10—20mm, 20—30mm, 30—60mm）, their compression characteristics are analyzed by MTS 815.03 electron hydraulic servo machine and home-made compression testing-equipment, and its compression-time characteristics are obtained. The results show that the strata weight of the fracture-arch in the stope is the main source of the compression effect of the coal gangue in the mined-out area. The coal gangue deformation is about from beginning to the dense condition; the coal gangue compression curve assumes a logarithmic relationship; the test results agree well with the field data.