I can't agree with this more. Starting a family should not be taken lightly as some kind of whimsical selfish destiny.
When people have kids later, it's because they're not ready. Some people are never ready because they are needed more elsewhere. Family will consume all of your free time, money, and effort. It's a big sacrifice.
I think a lot of people are simply anxious and have trouble making long-term commitments, and conflate that feeling with this abstract idea of readiness. No one is "ready" to do something for the first time including child-rearing.