Do they really? I have some black slacks that are 10 year old hand-me-downs from a relative and they look fine. My dress shirt for multiple years was a $5 second hand purchase - and I got more compliments on that shirt than on the ones that I got new.
Most such things are more about insider cultural knowledge than actual money. Most “upper class” activities and styles can be done cheap or free, but you need to know how.