Money is a superpower, one of mankind's most potent inventions, and just like with anything of great power it is learning to restrain yourself, to recognize when it is and is not appropriate. Self-restraint in the use of great power is what makes the difference between good and evil.
With money, all actions take on a dimension of moral hazard. You pay for a thing, but you do not do the work, so you don't know how it is done and what effect the work has on the worker or on the world at large.
If the altruistic angle doesn't appeal, consider the selfish angle: money divorces you from the reality of an act, making you less able to do for yourself.
Like the Amish, I suggest that you carefully consider what you pay for and why, rather than just decide based on whether you can pay for it.
Your question presupposes that the arrangement where certain people have to do shit jobs to survive, and others don't, is desirable. It works out fine for you individually, but the more charitable and humane arrangement is to spread the shit jobs around so no one bears the full brunt.
