Not a silly argument at all. Every time we say, "lets spend $x on y", it is fair to question whether that money can be used to feed and clothe... or defend.. or fix the infrastructure, etc.
Of course it can, there's no reason it couldn't. That doesn't make it a proper challenge. One would be "instead of $4B, $X should be invested on this, because of such and such". X == 0 without details is a big red flag.
You could make this argument for literally anything that isn't feeding, clothing or taking care of people, which means it is a silly argument.