"People of color" is the modern "black" (African heritage), and excludes dark skin colors from other locales. "Brown people", weird as it sounds as a phrase, is broader and includes dark-skinned groups like Middle-Eastern, but I think arbitrarily excludes Asian.
I've generally heard it as either "non-white", "non-white, with emphasis on 'brown' skin tones", or "non-white and not covered by a more specific group" in common real world use, depending on the context.
