If a computer cannot have infinite memory, then it cannot go beyond discrete finite automata (DFA) itself. Undecidable problems are several steps more complex than what a DFA can solve. In other words, the theoretical capability of a computer with finite memory is way less than a computer that can solve all but undecidable problems.
That's complicated. While a computer cannot, strictly speaking, have an infinite amount of memory (unless my understanding of physics is off-kilter, which is entirely plausible), computers do have an arbitrary amount of memory. Put another way, I don't know how much memory any given computer can access; for any computer you give me, I could imagine a computer with more memory. (Again, physics intercedes, so this is also something of a thought experiment.) So you can think of Turing-completeness as the limit of what a computer can do as you increase its memory.
To illustrate this fairly viscerally, a modern computer can access the internet and use an obscene amount of memory which will only go up in the future. We don't know how far up it will go, so modelling it as infinite makes sense.
Also, as another thought experiment, what if the universe is infinite? And what if it contains an infinite--or at least quickly expanding--amount of matter? Then a computer could effectively have infinite memory. Unfortunately, I really can't comment because I know even less about physics than I do about math :P.
Modeling it as infinite does not make any more sense than modeling it is having 1 googol^googol^googol memory cells, clearly above the physical limit, and therefore any differences between the two models are artifacts irrelevant to anything human civilization will ever encounter.
