I think it's the snow that really gets people. It just interferes with everyday life in a way heat doesn't (ie, limiting car usage and driving safety until it's cleared). This mostly applies to car-oriented lifestyles though - since walking in high heat vs snowy or icy conditions are both quite risky.
Also, there is actually a pretty strong correlation between shoveling snow and heart attacks in older folks: https://www.health.harvard.edu/blog/can-shoveling-snow-put-y... . So it definitely makes sense to relocate if that is a factor.