What if you combine this model with how Mozilla handles trademarks? Firefox is 100% open source, but if you distribute your own binaries, you can't use the name "Firefox" or the logo. That arrangement would seem to address your concerns.
That would help some of them, although it still cuts you out of potentially great feedback from your users (if you care about that.) Instead, users will file their bugs/raise their concerns/send their suggestions to Icestoat instead of your product.
You could, of course, watch that bugtracker (if they have one and if it is public), but it seems to me still to be counter to the ethos of releasing open-source software.
