Imagine you have a gene that will make half of your kids very strong, but unable to reproduce. Because of the tendency to protect the familly, your strong kids might help their sibilings to survive and reproduce, propagating their genes.
Another example is the ants. A gene that gives workers good abilities will help the hole colony, even if the worker themself do not reproduce.
It's better to think of a hive of ants/bees/etc. as a single organism. The individuals are cells. Things make much more sense in that context.
You're previous point, though, is spot on. In fact, I think we have genes just like that and not with regard to just strength. You might have a brother who becomes very famous because he's bright/sociable/etc. and your business prospers in the process, allowing you to have more kids. He wins that way, although indirectly. I think this happens a lot.
That's why it's such a pain to talk about these things in such simplistic terms ... life is just so much more complex and frankly, the evolutionary process is so brutal that there really aren't any losing strategies around. You can bet if somebody is doing something, they're benefiting somehow ... we just don't know how yet.
Ants have a intrinsic instinct to help the whole colony mostly because they are related the the entire colony.
I may have an evolutionary interest in supporting my nieces since they are my sister's children. However, I probably don't have an evolutionary interest in supporting my nephew since he is my wife's sister's child. Biologically I have zero relation to him.
Another example is the ants. A gene that gives workers good abilities will help the hole colony, even if the worker themself do not reproduce.