Of course you can be punished for exercising your rights. There is no better example than free speech: you're free to say whatever you want, and your customers are free to leave in response. The considerations of many contracts include some form of restricting your rights in return for something.
IANAL either, in case that wasn't obvious.