Ever been inside an actual castle? The lord's family was lucky if they had one or two private rooms. Most castles are tiny compared to their presentation in the media.
Aside from things like 18th-century manor houses - a tiny proportion of any country's population, and absolutely stuffed with working-class servants - the lack of motorized transit meant that upper-class neighbourhoods were usually right in the centre of the city, within walking distance of the poorer areas. You really just have to look at NYC or London to see that. 19th century mansions are not that far from the historically poor areas. And even those people who did own manor houses usually had a townhouse. It wasn't until trains, subways, and automobiles arrived in the mid-20th century that commuting between the business centre and exclusive suburbs became possible.
