Yeah, me too. I spent a couple of years of my childhood living in a trailer while attending a parochial school (because the local public schools sucked.)
I had a similar childhood.
I didn't live in a trailer, but in bad apartment complexes. The local public schools sucked, so my parents sent me to a parochial school with a very affluent student population. My parents didn't make much money, so I got to go for free. I always felt different there for some reason, but I didn't realize why until later: I was the "poor" kid.
I remember a friend's mother giving me a box of his old clothes once. When I was wearing a shirt of his the next day and told him about it, he got very pissed off.
I was usually the only student in my classes who got free lunches. I was too young to be embarrassed, but I think I was treated differently because of it.
So I think being the poor kid can have a stigma, but I still had sleepovers.