On paper, Python is not the right tool for the job. Both because of its bad performance characteristic and because it’s so forgiving/flexible/dynamic , it’s tough to maintain large Python codebases with many engineers.
At Google there is some essay that Python should be avoided for large projects.
But then there’s the reality that YouTube was written in Python. Instagram is a Django app. Pinterest serves 450M monthly users as a Python app. As far as I know Python was a key language for the backend of some other huge web scale products like Lyft, Uber, and Robinhood.
There’s this interesting dissonance where all the second year CS students and their professors agree it’s the wrong tool for the job yet the most successful products in the world did it anyway.
I guess you could interpret that to mean all these people building these products made a bad choice that succeeded despite using Python but I’d interpret it as another instance of Worse is Better. Just like Linus was told monolithic kernels were the wrong tool for the job but we’re all running Linux anyway.
Sometimes all these “best practices” are just not how things work in reality. In reality Python is a mission critical language in many massively important projects and it’s performance characteristics matter a ton and efforts to improve them should be lauded rather than scrutinized.
>the most successful products in the world did it anyway
A few successful projects in the world did it. There's likely far more successful products that didn't use it.
The key metric along this line is how often each language allows success to some level and how often they fail (especially when due to the choice of language).
> Instagram has one billion monthly users generating $7 billion a year.
Doesn't Instagram serve mostly static content that's put together in an appealing way by mobile apps? I'd figure Instagram's CDN has far more impact than whatever Python code it's running somewhere in it's entrails.
Cargo cult approaches to tech stacks don't define quality.
> But then there’s the reality that YouTube was written in Python. Instagram is a Django app. Pinterest serves 450M monthly users as a Python app. As far as I know Python was a key language for the backend of some other huge web scale products like Lyft, Uber, and Robinhood.
All those namedrops mean and matter nothing. Hacking together proof of concepts is a time honoured tradition, as is pushing to production hacky code that's badly stiched up. Who knows if there was any technical analysis to pick Python over any alternative? Who knows how much additional engineering work and additional resources was required to keep that Python code from breaking apart in production? I mean, Python always figured very low in webapp framework benchmarks. Did that changed just because <trendy company> claims it used Python?
Also serving a lot of monthly users says nothing about a tech stack. It says a lot about the engineering that went into developing the platform. If a webapp is architected so that it can scale well to meet it's real world demand, even after paying a premium for the poor choice of tech stack some guy who is no longer around made in the past for god knows what reason, what would that say about the tech stack?
If your goal is to actually ship product then this matters a lot. Many of us have dealt with folks spinning endlesslessly on "technical analysis" when just moving forward with something like python would be fine. Facebook is PHP.
I'm actually cautious now when folks are focused too much on the tech and tech analysis instead of product / users / client need.
> Why look at results when you can look at analysis!
The problem with this mindless cargo culting around frameworks and tech stacks is that these cultists look at business results and somehow believe that they have anything to do with arbitrary choices regarding tech stacks.
It's like these guys look at professional athletes winning, and process to claim the wins are due to the brand of shoes they are using, even though the athlete had no say about the choice and was forced to wear what was handed over to him.
I don't think "I could use tool X for job Y" implies "X was the right tool for job Y". You could commute with a truck to your workplace 300 feet away for 50 years straight and I would still argue you probably used the wrong tool for the job. "Wrong tool" doesn't imply "it is impossible to do this", it just means "there are better options".
The main thing is that Python is often in the top 10 choices for almost every problem on top of being insanely easy to learn and write; it also doesn't fragment its community - its standard library is so ridiculously large there are very few fault lines to break upon.
It's rarely the best choice, or even the fifth best. But if it's OK at a dozen things, then it makes it all but impossible to ignore. The fact that it sucks to write a GUI in is fine as long as I can put some basic "go" buttons and text boxes in front of a web scraper.
Depends on the margins you have to work with. For high-margin businesses, I agree, the tech stack isn’t crucial to the health of the business. But for low-margin businesses for which compute is a significant cost center, tuning a tech stack to be more cost-efficient can make you a hero.
You should read about MySpace and "Samy is my hero". Or Google vs. AltaVista. Or GeoWorks vs. Microsoft Windows. Or Yahoo Mail and the medireview problem. Or when Danger lost everybody's data. Or THERAC-25. Or Knight Capital's bug.
On the other hand, for many years a lot of Amazon's back office processes were written in Elisp.
With enough thrust you can get a pig to fly but effort can only compensate for bad technical decisions up to a point.
> Shel wrote Mailman [not the Python mailing list manager, an Amazon-internal application] in C, and Customer Service wrapped it in Lisp. Emacs-Lisp. You don't know what Mailman is. Not unless you're a longtime Amazon employee, probably non-technical, and you've had to make our customers happy. Not indirectly, because some bullshit feature you wrote broke (because it was in C++) and pissed off our customers, so you had to go and fix it to restore happiness. No, I mean directly; i.e., you had to talk to them. Our lovely, illiterate, eloquent, well-meaning, hopeful, confused, helpful, angry, happy customers, the real ones, the ones buying stuff from us, our customers. Then you know Mailman.
> Mailman was the Customer Service customer-email processing application for ... four, five years? A long time, anyway. It was written in Emacs. Everyone loved it.
> People still love it. To this very day, I still have to listen to long stories from our non-technical folks about how much they miss Mailman. I'm not shitting you. Last Christmas I was at an Amazon party, some party I have no idea how I got invited to, filled with business people, all of them much prettier and more charming than me and the folks I work with here in the Furnace, the Boiler Room of Amazon. Four young women found out I was in Customer Service, cornered me, and talked for fifteen minutes about how much they missed Mailman and Emacs, and how Arizona (the JSP replacement we'd spent years developing) still just wasn't doing it for them.
> It was truly surreal. I think they may have spiked the eggnog.
AFAIK, mailman was the only thing ever wrapped in this way.
I wrote all the rest of the back-office utilities (at least, the initial versions of them), and I have never come across any indication that they got "wrapped" in anything else (they did, no doubt, evolve and mutate in something utterly different over time).
Yegge's quote is also slightly inaccurate in that mailman, like all early amzn software, was written in C++. Shel and I just chose not use very much of the (rather limited) palette of C++ syntax & semantics.
> There’s this interesting dissonance where all the second year CS students and their professors agree it’s the wrong tool for the job yet the most successful products in the world did it anyway.
> I guess you could interpret that to mean all these people building these products made a bad choice that succeeded despite using Python but I’d interpret it as another instance of Worse is Better. Just like Linus was told monolithic kernels were the wrong tool for the job but we’re all running Linux anyway.
This isn't the correct perspective or take away. The 'tool' for the job when you're talking about building/scaling a website changes over time as the business requirements shift. When you're trying to find market fit, iterating quickly using 'RAD' style tools is what you need to be doing. Once you've found that fit and you need to scale, those tools will need to be replaced by things that are capable of scaling accordingly.
Evaluating this binary right choice / wrong choice only makes sense when qualified with a point in time and or scale.
YouTube video processing uses C++. It also uses Go and Java along with Python.
PInterest makes heavy use of Erlang for scaling.The rate-limiting system for Pinterest’s API and Ads API is written in Elixir and responds faster than its predecessor.
Takeaway: Basically you need to either build your own PythonVM/CPython fork for better Python performance or use another language for the parts that needs to scale or run fast.
I think people look ahead. They see how the app evolves (from A to B) and then claim "X not good". Where as they do not judge the flexibility of X as a tool to move from A to B. Typically they look at B and make claims from that perspective.
Those companies that succeed in python usually have a long path and python was never successfully removed and most likely attempts were made. The PL economics is often stickyness and its not easy to propose absolute measure.
We might be running Linux, yet it has so many virtualization layers on cloud infrastructure, user space stacks to workaround switching to kernel, with microservices for everything, that it is effectively a monolithic kernel being bended into a microkernel one.
Same thing with Python, those business succeed despite Python, and when they grew it was time to port the code into something else, or spend herculean efforts into the next Python JIT.
+1. Languages that are general purpose get used for everything. Perl for the web. Python for builds. Scala transpiled for web.
Portability has many solutions that are good enough - often only bad because they result in second order issues which are themselves solvable with limited pain. Being able to scale software further without having to solve difficult distributed systems problems is of value.
At Google there is some essay that Python should be avoided for large projects.
But then there’s the reality that YouTube was written in Python. Instagram is a Django app. Pinterest serves 450M monthly users as a Python app. As far as I know Python was a key language for the backend of some other huge web scale products like Lyft, Uber, and Robinhood.
There’s this interesting dissonance where all the second year CS students and their professors agree it’s the wrong tool for the job yet the most successful products in the world did it anyway.
I guess you could interpret that to mean all these people building these products made a bad choice that succeeded despite using Python but I’d interpret it as another instance of Worse is Better. Just like Linus was told monolithic kernels were the wrong tool for the job but we’re all running Linux anyway.
Sometimes all these “best practices” are just not how things work in reality. In reality Python is a mission critical language in many massively important projects and it’s performance characteristics matter a ton and efforts to improve them should be lauded rather than scrutinized.