You might feel different if you were a celebrity. I know one, and it's sad to see the curtains always drawn on the windows of the house, the paparazzi camped out in front, etc. I could see a nanny being hired and then threatening the celebrity with a tell-all book, or just cashing in on it anyway.
I don't understand your point. The condition of employment at many places, not just celebrities, is that you don't talk about it outside of work. You get well paid for this discretion. Do you think that is a violation of your rights? I don't, the moment you accept the money.
How about NDAs? Should they be unenforceable?
How about Secret Service agents protecting the President. Should they not have to zip their lips about what they see and hear about what goes on in the White House? Is that a violation of their rights to accept such conditions of employment?
Many disputes are settled with the terms being non-disclosed. Should those be unenforceable, too?