Well, I can do math, but I very rarely "think in mathematics." Rather, I approach most math problems linguistically, by essentially making up a little word problem in my head and reasoning through it. However, I know other people who approach math in a completely different way. They seem to have an intuitive "calculating sense," and only use language "after the fact" to record the result or explain it to others.
So in the same sense that I can use math without it being central to my thinking, it doesn't seem hard to believe that others can use language without it being central to their thinking.
So in the same sense that I can use math without it being central to my thinking, it doesn't seem hard to believe that others can use language without it being central to their thinking.