When I hear him talking about kids, it's almost invariably that they misbehave. Not that they're visiting X or Y with kids or had a funny talk or something. And the kids seem a bit too eager to play with anyone, as if no one spends much time with them. Does my brother only work late on holidays? To me it sounds they're managing their kids, not raising them. Maybe I'm bad at judging people, but I don't see much joy in that family.
Anyway I think this is a bit sad. Ancient Greeks valued friendship more than love. Friends are often for life, while lovers are all too happy to switch when someone more attractive shows up.
It definitely sounds like you have something against your brother by the way you have portrayed him in a negative light. It sounds like you are competitive against him by constantly comparing his lifestyle versus yours.
You make him sound lonely. But he has a wife and kids. Most people get less involved in their friendship group after starting a family