Going for drinks after work with your male colleagues, but not the women, is most likely a net positive for your career. Very rarely is anybody fired for such an offense as not hanging out with women after work, but building a closer relationship with the other men you work with will pay dividends.
