could it just be that if the 'privileged user' upvoted or downvoted it, it would be considered an endorsement whereas not touching it would leave it pending?
But that's the problem. Upvoting and approving/moderating are different things. Upvoting is a subjective exercise, whereby you signal substantive agreement. Approving/moderating is an objective exercise, whereby you signal that a comment meets minimal community standards of tone and non-triviality. Conflating those will inhibit dissenting opinion.
That said, the direction you're implying might be an improvement. Rather than a separate Pending Comments list, mod-level users would be able to see pending comments in the context of the parent item/tree, but with some kind of adornment or different color. Pending comments will also have a 'tolerable'/'civil'/'endorse' link next to it, but no up/down vote buttons. Only after it's been deemed tolerable enough, will a pending comment become live/approved and have up/down buttons.
Indeed, that seems a much more sensible interaction-model to me. Take the absolute number of upvotes as the "endorsement value" of a post. Make all posts start at zero "endorsement", and require them to reach some N before they appear.