"I keep coming across statistics and CS PhDs who now work for Twitter, the New York Times, and industry research labs (AT&T, Microsoft). Why isn't an industry job a viable option?"
Because (speaking as one of those people), a PhD is total overkill for nearly all industry jobs, and it costs a lot more to get one. It also probably works against you in most parts of the tech industry, where there's a surprising amount of blind opposition to anyone with a doctorate.
Finally, remember people with PhDs who work in industry have made a difficult, conscious decision to abandon the academic life. It's not the expected outcome, and there's an intense cultural pressure not to leave the ivory tower.
"It also probably works against you in most parts of the tech industry, where there's a surprising amount of blind opposition to anyone with a doctorate."
I didn't encounter any doctorate-opposition per se. I think it's more opportunity cost issue.
Tech industry is quite meritocratic. Problem for many people with PhDs is that this is the only thing they can show after many years spent hidden in academia, working on esoteric things.
If you keep up your real world skills during graduate school, I believe nobody is going to hold your degree against you.
At least that was my experience (and experience of my classmates from graduate school).
We did a lot of nitty-gritty software engineering during graduate school (ideas from our papers had to be implemented and integrated into bigger projects, that's how funding pipeline worked).
Also it helps not to act smug about your degree - industry is full of very smart people who didn't even go to university.
Let's be clear: I'm not complaining, nor am I talking about personal experiences of discrimination -- I don't know if my resume has ever been circular-filed because of my degree. But since I left academia I have been surprised by the number of people who have explicitly told me that they consider a PhD to be a black mark on a resume. I think a lot of people have had one or two bad experiences interviewing/hiring PhDs, and they associate the degree with the incompetence because it's so rare to interview someone with a doctorate.
For what it's worth, I don't find the tech industry to be more or less meritocratic than any other -- we certainly like to pretend that our hiring methods are hyper-objective, but I've seen lots of hiring decisions that just boil down to opinion and intuition. Non-meritocratic things like pedigree and 'who you know' matter a lot, even amongst engineers.
Fully agree with what you said - there is a lot of sampling bias because of relative rarity of PhDs (few bad apples can completely color expectations).
It was quite a surprise for me when doing a summer program at major US corporation and everybody was going gaga because our group had many PhDs.
In academia, everybody has doctorate, so degree in itself doesn't really confer any additional signal.
In industry, people take it as a signal even when it is not (person matters more than degree, the same person would be hireable / not-hireable whether having or not having degree).
But anyways, you wouldn't want to work at places / for people which can't / don't take such things into account.
That's why I mentioned meritocracy - it's nicer to work at places where it matters only if you can get the job done, not your degree / pedigree / who-you-know.
But yeah, human nature, hard to fight against, we all like signaling (it's useful heuristics after all).
Because (speaking as one of those people), a PhD is total overkill for nearly all industry jobs, and it costs a lot more to get one. It also probably works against you in most parts of the tech industry, where there's a surprising amount of blind opposition to anyone with a doctorate.
Finally, remember people with PhDs who work in industry have made a difficult, conscious decision to abandon the academic life. It's not the expected outcome, and there's an intense cultural pressure not to leave the ivory tower.