And yet, the population in many western countries is cratering. Right now there is a housing crunch in some places, but once all the baby boomers die then there should be plenty of houses for the smaller number of children being born. And as a bonus the housing prices should tank since the supply should far outstrip the demand.
It seems, US population is not cratering. According to this population projection, it will continue to increase all the way to 2060 and beyond: https://www2.gwu.edu/~forcpgm/Ortman.pdf .