I totally agree with this. I even had the SSH firewall rule of my Mumble voice server for my WoW guild set to only allow my home connection. One downside is if I'm ever given a different IP address by my ISP, I'd have to go to work and change it. Minor inconvenience but this isn't a crucial service.
