In my philosophy, it is close to axiomatic that free exchanges leave both parties better off.
In externally quantifiable terms, that's something I think, too. However, there are so many personal examples where it's not really true that I'm beginning to wonder about it. For example, when a pretty girl comes up to me (as in the linked blog) and asks me for something I would be reluctant to provide her with, the very act of asking me has already made me worse off. At this point, it's true, I can make myself better off than I now am by doing whichever I would really prefer, but I would be better off still had she not asked.
Because the outcomes available to me have reduced from the three: "continue as before", "disappoint [person] in front me", or "feel upset about giving someone X when I would have preferred not to", to only the last two. The first option, which could well have been the best outcome in my opinion, has been removed by the act of asking.
Edit: changed the other individual to "person" to be clearer.
If you think your refusal makes them sad, do you think you have a kind of telepathy? Isn't believing you control the outside world with your mind an indication of schizophrenia?
It's an illustration by absurdity that your thoughts are limiting you and making you unhappy, and that there are other ways to think which are still compatible with being a good person and a moral person.
For as long as you hold in your head the belief that your refusal is what hurts people then you are going to be unhappy about being asked questions.
If instead you held the belief that their expectations and demands in their head is what hurts them, then you would be free to accept or refuse politely and in good conscience, and still be able to be a person who "considers how he affects others".
The fact that you cannot consider saying no is what makes a joke out of your comment "I don't want to be a person who doesn't care about his effect on others". At the moment you are forced to accept regardless of whether you care or not. Caring doesn't come into it, they ask and you force yourself to accept, you don't get to choose or consider.
If you didn't force yourself to accept because you were no longer afraid that it was you hurting them, then you would be free to consider their request unpressured and respond however you felt like at the time, but still respond based on your morals and so on.
At the moment you say you are someone who cares, but you act in a way that you ignore your consideration and decisions and marginalise your caring making it subservient to a fixed instruction. You say you don't want to be someone who doesn't care, but changing that part of you would free you up to be someone who can care and isn't forced to act one way or the other.
Well the pretty girl wasn't going to sleep with you anyway...perhaps you just need more practice talking to pretty girls until you realize they're mostly just regular people, too. I say this as someone who used to have the same problem.
In externally quantifiable terms, that's something I think, too. However, there are so many personal examples where it's not really true that I'm beginning to wonder about it. For example, when a pretty girl comes up to me (as in the linked blog) and asks me for something I would be reluctant to provide her with, the very act of asking me has already made me worse off. At this point, it's true, I can make myself better off than I now am by doing whichever I would really prefer, but I would be better off still had she not asked.