The answer to that may actually be quite simple: The keyboard app is the one that should be allowed to read from the clipboard. Now can a keyboard spy on you clipboard contents? Sure. But it can also spy on all your keystrokes, so the trust must already be there.
That assumes that clipboard can only contain text, which I'm not sure whether is so on Android, but on all major desktop UI systems clipboard can not only contain arbitrary data, but even allows for negotiation of supported formats between source and destination.
