But you do not need to be passionate about the right things from the beginning. For one thing, it is possible to move into a field late in life and make enormous advances. Fermat provided a brilliant example of this.
Of course, you could say Fermat was an outlier, but I suspect that anyone who is truly, deeply passionate about something is an outlier already. Most people go through their lives with little passion at all.
I was passionate about zoology during some of the formative years of my life.
I realized later that unless I become the next Darwin, I will never be happy with my career choice because I will be doomed to working for a large part of my life for people less intelligent than myself and that I will not grow as a person because of that.
Of course, you could say Fermat was an outlier, but I suspect that anyone who is truly, deeply passionate about something is an outlier already. Most people go through their lives with little passion at all.