I doubt you will find any Olympic Games in the last 40 sears that did not attract people being opposed to them. In the end, it binds resources some thing may be used more efficiently elsewhere, as if the money is gone after spending it.
In this case in particular, what started as mostly a joke got fueled by the cost paid for postponing the event, coupled with the eventual penalties that would apply if the event was canceled altogether.
There was a heated moment the first time the gov had to decide wether to swallow the sunk costs and just cancel everything, or only postpone, hoping the situation would be better after a few months.
The first set of damages have already be paid, so now whatever the course of events, Olympics will be a pretty bitter subject for most people I think.
Edit: postponing was not penalties per se, more contracting and maintenance costs. Fixed the wording,
