That is a reasonable response. I would counter by saying that I see that as a justification for why they [government, ISP, whoever] should be allowed make demands about the use of the physical commons, but isn't a reason why they should exercise this conceded right.
For example: I, a hypothetical bar owner, have a right to ban silly hats in my bar. Why? Because I own it. However that's not a reason that I should ban silly hats. Just saying "I'm the owner, so I can." doesn't actually explain why I should.
I do also disagree that the government specifically is entitled to restrict privacy on the internet because much of the infrastructure is owned or otherwise controlled by them. The government owns nearly all roads, yet while using that infrastructure I still enjoy certain privacies. For example, if a cop pulls me over and asks me where I am going or where I have been, I have no obligation to answer him.
"Yes, yes, you have the right not be subjected to unreasonable search and seizure, but not if you're walking on the public sidewalk."
The key here is that the government is not like a private land owner. That's why government workers have so much leeway in criticizing their employers, and why you're allowed to protest on government land, and why schools can't have daily prayers even though they want to. The government simply isn't a private actor enforcing rules over the stuff it owns; we own the commons, and the government simply administers them according to some strict rules. Unlike a private land owner, it can't say, "if you don't like the way I do things, take your business elsewhere." Otherwise you get "You can vote for whomever you want, but since we own all these voting booths, you can only use them to vote for the incumbent."
If the government can pursue any arbitrary policy just by virtue of "owning" some infrastructure, the whole constitutional democracy thing gets circumvented.
Edit: changed "to privacy" to "not to be subjected to unreasonable search and seizure" to avoid confusion.
A hypothetical bar owner could get away with banning silly hats. We can defend that with assertions about ownership of property and the privileges that gives somebody... that isn't particularly problematic within reason. Governments though? They need to play by a different, stricter, set of rules. "Ownership of infrastructure" should not be accepted as a defense of a government banning silly hats on their sidewalks.
Governments operate in a privileged space where they are permitted to do many things that individuals and companies are not allowed to do (as a quick example, they can levying taxes against the general population). That has to come at a cost though; they aren't allowed to do things that individuals and companies are permitted to do (as a quick US-centric example, they cannot endorse and support a particular religion).
For this reason, comparisons and analogies between what governments and individuals/companies can do are very frequently worthless at best. These sort of comparisons are just unavoidably apples and oranges.
For example: I, a hypothetical bar owner, have a right to ban silly hats in my bar. Why? Because I own it. However that's not a reason that I should ban silly hats. Just saying "I'm the owner, so I can." doesn't actually explain why I should.
I do also disagree that the government specifically is entitled to restrict privacy on the internet because much of the infrastructure is owned or otherwise controlled by them. The government owns nearly all roads, yet while using that infrastructure I still enjoy certain privacies. For example, if a cop pulls me over and asks me where I am going or where I have been, I have no obligation to answer him.