Most people afraid of “math” only have exposure to the second kind. I figure it has more to do with finding those type of mechanical operations boring (they are), and hence all of maths must be equally boring. If you’re a software developer, enjoy abstractions, programming language theory, and solving problems there is no way you would find math of the first sort boring. Alas, in high school pretty much all the math is of the boring mechanical kind. At least mine was, anyway. Even the first exposure to non-rigorous calculus is of the shitty plug-and-chug variety.
It's hard to go through life without developing a modest understanding of arithmetic and reading. You wouldn't even be able to check your receipts if school started with Peano's axioms. Consequently, numbers come first, as they have for millennia now. Which makes sense, because maths developed out of intuitive arithmetic.
As a matter of fact, my parents complain they never understood maths, because they were taught algebraic rules by rote. a * b = b * a. They can recite that, they just don't have a clue what to do with it.
Teachers just aren't good, and certainly not at maths. It's a struggle to teach billions the basic skills. There is no solution.
I also don't know any mathematician or physicist bad at arithmetic. Quite a few of them seem to delight in numbers and arithmetic puzzles.
So I really don't see how to avoid numbers and boring school work, nor how the former could have a positive effect.