Programs written in programming languages aren't guaranteed to terminate. Computable numbers, however, are all real numbers that can be represented to arbitrary precision by a computation that terminates. Wouldn't that mean we don't need a fully fledged programming language to have the capability to represent all computable numbers?
Somewhere in there is the idea we just need a program that determines whether an arbitrary program terminates or not...
I can't tell how much of this is a joke, but, yes, it turns out you can't write such a programming language for exactly the reason you allude to: you can't tell whether a given number is computable in finite time, as you don't know whether your program will terminate.
I believe they call those "programming languages"