`dirs` is a shell built-in for the directory stack. If you supply it with -v, it will number the output. So the trick is just to make these line up, which is also really just using a feature of `cd` that extracts an entry from the directory stack at position n. My aliases look like this:
alias dirs='dirs -v'
alias '1'='cd -'
alias '2'='cd -2'
alias '3'='cd -3'
etc.
Doesn't seem like it would make a big difference, but it's very fluid and adopts well to whatever "you happen to working on in the moment."