Not really. When I work, I work. And I get to spend a lot more time with my family than I would otherwise. I have plenty of interaction with customers via email, skype etc. Also I hang out online with other entrepreneurs, who are generally very helpful and supportive. Its not quite the same as face-to-face interaction. And I guess I don't meet as many new people as I would working in an office. But I don't have to put up with people I don't like and I certainly don't miss office politics!
If you are very extrovert, it might not be for you. But how many extrovert programmers do you know? ;0)
Do you miss the human interaction one gets at the office?