The issue is any antibiotic that could be used to cure humans should be used for nothing but curing humans. Every use of it rolls the dice that the bacteria will evolve an immunity to that drug. There isn't any "safe" use. Bacteria will eventually evolve and we can only reduce the amount of evolutionary pressure being applied.
So in short, yes the problem is giving it to animals.
But we derive real utility from using antibiotics with animals. First in preventing blights in our food supply (which has been the downfall of many nations of antiquity) and second in controlling diseases that can jump to humans.
If there is no safe use, then what matters is risk vs. reward, right? There is real utility (reward) to using antibiotics on animals, and I don't see how we can simply blanket rule it out.
So in short, yes the problem is giving it to animals.