I have 2 small children as well. Both with autism. My wife and I have no problem leaving the children with one or the other. We don't do it frequently, but we do still enjoy our own hobbies. I love knowing that she can go spend a day out photographing with her friends, doing what she wants. And I love the day I get to spend with my kids, just me and them.
Being understanding doesn't mean you have to share in everything, together every moment.
I also have a child with autism and I think that factors into it. Both in that it can be much easier for an auteur to handle one on one situations, and because it can be really draining dealing with the negative aspects of autism and I think it's healthy to have some respite from it (maybe somewhat true with all kids but much more so with social/developmental disorders)
I guess it depends on the children and possibly on how well you can handle them. It's certainly pretty hard for our parent/child combination, I couldn't imagine spending even a full day alone with them, and my wife is always exhausted when I have to go on a business trip. I'm optimistic that it'll get better when they stop being toddlers though :)
Being understanding doesn't mean you have to share in everything, together every moment.