Certainly not. The ultra-cheap stuff is cheap throughout the production pipeline. Costs are cut wherever possible: wages, tools, work environment, materials, packaging...
The "poor people" won't benefit from most of the branded product's premium, of course. But it's more likely that it's been produced by a subcontractor that at least respects local laws like minimum wage, construction standards and regulations concerning toxic materials.
Being blunt, how do we know? For example, when the buildings collapsed back in 2013, the discount chains selling unbranded clothes like Primark and Loblaw pledge to offer compensation, while Benneton tried to hide their relationship with the manufacturer.
As it ever been studied if the work conditions of the workers making branded stuff are generally better than their counterparts working on cheaper clothes?