That's what "collaborator" means in Github parlance. If you don't want people to have this permission, don't make them collaborators. If you want people to be collaborators but work mostly on their own fork and send pull requests, ask them.
From your description it sounds like you're talking about self-hosted repo's not Github. For that you have a couple of options, you can use a gatekeeper such as Gitosis and assign permissions. Or, simply have people pull through a read only mechanism (like git daemon) rather than giving them ssh or other writeable access.
I want them to fork and "pull request", but it seems they can just push back into my master at will.
Maybe I'm looking at it the wrong way.