You are overlooking a crucial fact: very often people who want to use Lisp are not allowed to by their management precisely because "no one uses it" and so the experiment never gets done.
I was the lead engineer on the first release of AdWords. I wanted to write it in Lisp, but I was not allowed to, being forced against my strenuous objections to to it in Java. So now AdWords is a Java success story rather than the Lisp success story it might have been not because Java is better (it certainly wasn't -- it was a disaster) but because my boss issued an edict.
This has happened to me many times in my career. The one time I was actually allowed to use Lisp in a project that I did not have direct control over (the DS1 remote agent) it was an overwhelming technical success. In fact, management tried and failed to get the software rewritten in C++, so this was actually a controlled experiment.
If that were the (only) issue, we ought to see Lisp used more in startups than in established businesses, because in startups, you don't have some manager who doesn't know Lisp and is managing to minimize is perceived (not necessarily actual) technical risk.
But in fact I'm not sure that we see more Lisp use in startups, either. That could be because they aren't taught it in school. (Many startup founders haven't had the chance to pick it up on the job. For that matter, many programmers haven't had the chance to pick it up on the job.)
On the controlled experiment: Why was that? I mean, it has to be possible to rewrite anything in C++ (Turing complete, and all that - but possibly at the price of Greenspun's Tenth Law). Was it because the people who tried didn't know anything about Lisp? (Were they working from the source, or from the spec, or from the program documentation?) Were they just not as good programmers as the Lisp programmers (and don't say "if they were as good, they would have been using Lisp" - that wasn't their assignment). Did they have less time, less resources? Why didn't/couldn't they do it?
It's all a viscous cycle going back to AI winter in the late 80's: DARPA stopped funding AI work, which meant that funding for Lisp work dried up, which meant that fewer people used it, which meant that fewer people learned it. Now, 30 years later, hardly anyone uses it, at least not directly. But it keeps getting re-invented again and again. You can't avoid re-inventing Lisp because it's part of the fundamental physics of computing. That is my frustration. People say that Lisp sucks, and then they proceed to re-invent it badly not even being aware that that is what they are doing.
As to why the re-implementation of the Remote Agent (it wasn't the whole thing BTW, just the planner) in C++ failed, it was a combination of factors. This was twenty years ago (holy cow!) and C++ compilers were nowhere near as mature then as they are now. There was only one compiler available for the flight hardware, and it was pretty unstable. But mainly it was Greenspun's tenth: the application did a lot of dynamic allocation (it was an AI search algorithms) which is something C++ is particularly not well suited for. They essentially had to re-implement Lisp in C++, and that was hard. Of course it would have been possible given more time, but we didn't have more time. The Lisp code was working, so that's what we flew.
Supposing the AI winter never happened, what is your perception of where Lisp would be now? Dominant? Large niche? Small niche, but larger than present?
I have no idea, and it doesn't really matter because we can't go back. I really don't want to dwell on the past, except insofar as we can learn lessons that inform the future. The point I really want to make is that Lisp is still a viable option today (Common Lisp and Clojure in particular) and I would like to see people give it a fair shake going forward.
I was the lead engineer on the first release of AdWords. I wanted to write it in Lisp, but I was not allowed to, being forced against my strenuous objections to to it in Java. So now AdWords is a Java success story rather than the Lisp success story it might have been not because Java is better (it certainly wasn't -- it was a disaster) but because my boss issued an edict.
This has happened to me many times in my career. The one time I was actually allowed to use Lisp in a project that I did not have direct control over (the DS1 remote agent) it was an overwhelming technical success. In fact, management tried and failed to get the software rewritten in C++, so this was actually a controlled experiment.