Well, I'm OP, and as fate would have it, I'm actually from Detroit (born and raised).
If you do go, please make sure you have a tour guide. Downtown is safe enough, but if you decide to see the "real" city, you should know where you are going before you go there.
Edit: Just realized you made a comment earlier. Well, as we are discussing now -- I grew up in a low income area in inner-city Detroit and I am a person of color. I never really considered myself as privileged. It's all about perspective.