You owe people politeness, but I don't think that you owe them a justification for not wanting to do things. If it's rude to say no, it's rude to ask.