With just this, it would be fairly easy to learn how many votes translate to what color, and you're back at square one. Maybe color as some function of upvotes on the comment and proportion of all upvotes (maybe only those comments at the same level in that thread)?