You're right, that is annoying. Thanks for the suggestion. Here's what we're going to do. On hover on the Circles list, we'll swap out the member count for an Edit link:
Oh wow, that's great! I'm currently trying to get some of my friends on board so we can use this. I think they'd appreciate the interface and the privacy - which they all find quite jarring about Google+ and Facebook respectively!
https://img.skitch.com/20120517-gg1dd9n3ecy5882gd8335tap44.j...