SRBs were made in Utah by Thiokol and sent by rail in sections. The limiting factor (choke point) was a single mountain tunnel, which was built a long time ago and was influenced by the winners of the rail gauge wars.
If you want to be confrontational skeptic to seem original, that's all you.
You can see my nearby thread for a more detailed look at the claim.
The tl;dr: the "limiting factor" is a tunnel width of 18', against a standard US loading gauge of 10'6", and a standard US track gauge of 4'8½" ... and the SRBs are 12' in diameter, which doesn't seem clearly limited by any of those factors whatsoever.
There could also be switching gear, signage, etc internal to the tunnel they had to deal with - if the claim is even valid in terms of this being the spurce of restriction on booster size.
A story that's passed around a million times whose truth content is damn close to nothing?