I like Tom a lot but my own theory as to why he is in The Lord of the Rings is because Tolkien had already created him. You get a sense that the world-building Tolkien did began with a somewhat more fairytale-like world with characters like Tom Bombadil. First The Hobbit and then The Lord of the Rings took the world up to something less childish, more ... Arthurian?
Tom was at that point a round peg in a square hole, but Tolkien shoehorned him in nonetheless.
The History of Middle Earth gives an almost unprecedented insight into how an author works. Which is fortunate, since Tolkien himself is an unprecedented writer.
He had a singular way of revising, almost pathologically, holding some key points utterly fixed and revamping everything else to make them fit. It's why he never finalized The Silmarillion, and why when his son finally did publish it, he discarded a lot of the work done post-Lord of the Rings.
I believe you are correct: Tom was one of those fixed points, for no reason he could explain. He later said that Tom was left as a deliberate mystery. Which is a bit disingenuous, but that's fine. Tolkien worked by instinct, and his works are extraordinary. If his instinct tells him that Tom needed to be in the narrative, that's fine by me.
If other readers are perplexed by it, they will hopefully forgive it long enough to get through the book and read it again. It's very much the kind of book that you won't fully understand in a single reading, if ever.
Probably the people who most need to understand why he is in there.