Of course. It’s not clear that the “real” or even “natural” numbers meaningfully correspond to any physical phenomenon. They’re just abstractions and tools for making predictions and reasoning about our experiences as humans, which don’t necessarily reflect how the world actually is.
However while there are no physical realizations of infinite sets, they're used all the time in math
So I guess mathematical entities do not need to be connected to physical elements