Where does one host such a service such that one's neighbors haven't put your subnet on e-mail block lists? I wasn't able to host e-mail from my home IP address; do spammers not pollute the waters at hosting companies? Can you buy an IP address that isn't blacklisted, and won't be part of a subnet block?
Since there is so much malware out there delivering spam from private computers, dynamic IP ranges (which is what you usually get at home) are part of every modern blacklist.
Yes, this is very unfortunate. It means that you'll either have to rent some server or virtual server (which makes sense if you want to run tons of other services there).
Or, you'll need some VPN (possibly in addition to an IPv6 tunnel), which is cheaper.
A third option is to rent an own server/vserver, but to keep the costs low by sharing it with friends.