At my university, the microbiology class did a test every year to see how well people wash their hands. They were instructed to was their hands in scolding water for four minutes with antibacterial soap. Turns out, even after that they still had more than half the bacteria left on their hands. This of course says nothing about whether you'll get sick.
EDIT: Also, don't try to pull any of this at a hospital. In those situations you can't afford to take chances (as in when others' health is at risk).
Same here except they do get washed with soap when I do the dishes. Why would you get sick if you don't wash your hands? Because they touch food? You have nothing to worry about, you can eat food from the ground and you won't get sick. Rather than worrying about dirtiness, worry about too old food (especially bread with fungus, I did become sick when I ate that, but I hear it's dangerous).
I think it's down to technique.
a) use the hottest water you can stand. With practice you can get it pretty hot.
b) scrub your hands together completely, vigorously, and for a good 20 to 30 seconds.