I'm sure colourblind people would be even more "bummed out" that your design was unusable. Since, as you go on to mentioned, you don't have to sacrifice aesthetics for the sake of accessibility, I don't see why accessible design would be a problem.
While I take your point that ramps in particular aren't always appropriate, that doesn't mean that accessible design in general isn't.
While I take your point that ramps in particular aren't always appropriate, that doesn't mean that accessible design in general isn't.