That's why when any boss asks for something completely irrelevant for the job, in my opinion the correct approach would be to ask for a documented request. i.e. "please, send me an official email with your request". If the boss then goes crazy, then reveals he had no good intentions from start, and yes, it's definitely hard to fight back without eventually get to that point of no return. If that's the case, you then better leave that place before getting fired, which has the side effect of showing that you're not that easy to manipulate. It's not just about self esteem and/or testosterone however: any time diverted from the job to other things could impact the project, and doing that during your time means there's your name stuck to the job at that time; if something goes wrong because of the time spent doing other stuff then it is your fault, so he better gives some good official explanation for demanding that extra work.
(edit: moved under the right parent post)