According to the theory of surface energy, the surface free energy parameters of styrene- butadiene-styrene (SBS), crumb rubber (CR) and composite crumb rubber (CCR) of three kinds of polymer modified asphalt before and after salt freezing cycle and limestone, basalt, granite were measured by contact angle measurement method. Adhesion work and spalling work of asphalt-aggregate system were calculated before and after salt freezing cycle, so as to determine the influence of salt freezing cycle on the adhesion and spalling properties of asphalt-aggregate system. The results show that the specific surface free energy variation of adhesion process and spalling process of asphalt and aggregate system are all negative before and after salt freezing cycle, explaining the process of adhesion and spalling of asphalt and aggregate are all spontaneous. With the increase of salt freezing cycles and salt concentration, the adhesion work of asphalt-aggregate system gradually decreases, and the spalling work gradually increases, adhesion and resistance to water damage ability are weakened gradually. When the asphalt type is the same, the adhesive work of asphalt and limestone is the largest, and the spalling work is the smallest, the adhesive work of the asphalt and granite is the smallest, and the spalling work is the largest. When the aggregate type is the same, the adhesive work of CCR modified asphalt and limestone is the largest, and the spalling work is the smallest, the adhesive work of the SBS modified asphalt and granite is the smallest, and the spalling work is the largest.