With JavaScript, these kinds of optimizations in an engine make sense due to the web being limited by it and thus speed is a huge factor. With Python, however, if a Python web framework is “too” slow, I would honestly say the problem is using Python at all for a web server. Python shines beautifully as a (somewhat) cross platform scripting language: file reading and writing, environment variables, simple implementations of basic utilities: sort, length, max, etc that would be cumbersome in C. The move of Python out of this and into practically everything is the issue and then we get led into rabbit holes such as this where since we are using Python, a dynamic scripting language, for things a second year computer science student should know are not “the right jobs for the tool.”
Instead of performance, I’d like to see more effort in portability, package management, and stability for Python because, essentially since it is often enterprise managed, juggling fifteen versions of Python where 3.8.x supports native collection typing annotations but we use 3.7.x, etc. is my biggest complaint. Also up there is pip and just the general mess of dependencies and lack of a lock file. Performance doesn’t even make the list.
This is not to discredit anyone’s work. There is a lot of excellent technical work and research done as discussed in the article. I just think honestly a lot of this effort is wasted on things low on the priority tree of Python.
On paper, Python is not the right tool for the job. Both because of its bad performance characteristic and because it’s so forgiving/flexible/dynamic , it’s tough to maintain large Python codebases with many engineers.
At Google there is some essay that Python should be avoided for large projects.
But then there’s the reality that YouTube was written in Python. Instagram is a Django app. Pinterest serves 450M monthly users as a Python app. As far as I know Python was a key language for the backend of some other huge web scale products like Lyft, Uber, and Robinhood.
There’s this interesting dissonance where all the second year CS students and their professors agree it’s the wrong tool for the job yet the most successful products in the world did it anyway.
I guess you could interpret that to mean all these people building these products made a bad choice that succeeded despite using Python but I’d interpret it as another instance of Worse is Better. Just like Linus was told monolithic kernels were the wrong tool for the job but we’re all running Linux anyway.
Sometimes all these “best practices” are just not how things work in reality. In reality Python is a mission critical language in many massively important projects and it’s performance characteristics matter a ton and efforts to improve them should be lauded rather than scrutinized.
>the most successful products in the world did it anyway
A few successful projects in the world did it. There's likely far more successful products that didn't use it.
The key metric along this line is how often each language allows success to some level and how often they fail (especially when due to the choice of language).
> Instagram has one billion monthly users generating $7 billion a year.
Doesn't Instagram serve mostly static content that's put together in an appealing way by mobile apps? I'd figure Instagram's CDN has far more impact than whatever Python code it's running somewhere in it's entrails.
Cargo cult approaches to tech stacks don't define quality.
> But then there’s the reality that YouTube was written in Python. Instagram is a Django app. Pinterest serves 450M monthly users as a Python app. As far as I know Python was a key language for the backend of some other huge web scale products like Lyft, Uber, and Robinhood.
All those namedrops mean and matter nothing. Hacking together proof of concepts is a time honoured tradition, as is pushing to production hacky code that's badly stiched up. Who knows if there was any technical analysis to pick Python over any alternative? Who knows how much additional engineering work and additional resources was required to keep that Python code from breaking apart in production? I mean, Python always figured very low in webapp framework benchmarks. Did that changed just because <trendy company> claims it used Python?
Also serving a lot of monthly users says nothing about a tech stack. It says a lot about the engineering that went into developing the platform. If a webapp is architected so that it can scale well to meet it's real world demand, even after paying a premium for the poor choice of tech stack some guy who is no longer around made in the past for god knows what reason, what would that say about the tech stack?
If your goal is to actually ship product then this matters a lot. Many of us have dealt with folks spinning endlesslessly on "technical analysis" when just moving forward with something like python would be fine. Facebook is PHP.
I'm actually cautious now when folks are focused too much on the tech and tech analysis instead of product / users / client need.
> Why look at results when you can look at analysis!
The problem with this mindless cargo culting around frameworks and tech stacks is that these cultists look at business results and somehow believe that they have anything to do with arbitrary choices regarding tech stacks.
It's like these guys look at professional athletes winning, and process to claim the wins are due to the brand of shoes they are using, even though the athlete had no say about the choice and was forced to wear what was handed over to him.
I don't think "I could use tool X for job Y" implies "X was the right tool for job Y". You could commute with a truck to your workplace 300 feet away for 50 years straight and I would still argue you probably used the wrong tool for the job. "Wrong tool" doesn't imply "it is impossible to do this", it just means "there are better options".
The main thing is that Python is often in the top 10 choices for almost every problem on top of being insanely easy to learn and write; it also doesn't fragment its community - its standard library is so ridiculously large there are very few fault lines to break upon.
It's rarely the best choice, or even the fifth best. But if it's OK at a dozen things, then it makes it all but impossible to ignore. The fact that it sucks to write a GUI in is fine as long as I can put some basic "go" buttons and text boxes in front of a web scraper.
Depends on the margins you have to work with. For high-margin businesses, I agree, the tech stack isn’t crucial to the health of the business. But for low-margin businesses for which compute is a significant cost center, tuning a tech stack to be more cost-efficient can make you a hero.
You should read about MySpace and "Samy is my hero". Or Google vs. AltaVista. Or GeoWorks vs. Microsoft Windows. Or Yahoo Mail and the medireview problem. Or when Danger lost everybody's data. Or THERAC-25. Or Knight Capital's bug.
On the other hand, for many years a lot of Amazon's back office processes were written in Elisp.
With enough thrust you can get a pig to fly but effort can only compensate for bad technical decisions up to a point.
> Shel wrote Mailman [not the Python mailing list manager, an Amazon-internal application] in C, and Customer Service wrapped it in Lisp. Emacs-Lisp. You don't know what Mailman is. Not unless you're a longtime Amazon employee, probably non-technical, and you've had to make our customers happy. Not indirectly, because some bullshit feature you wrote broke (because it was in C++) and pissed off our customers, so you had to go and fix it to restore happiness. No, I mean directly; i.e., you had to talk to them. Our lovely, illiterate, eloquent, well-meaning, hopeful, confused, helpful, angry, happy customers, the real ones, the ones buying stuff from us, our customers. Then you know Mailman.
> Mailman was the Customer Service customer-email processing application for ... four, five years? A long time, anyway. It was written in Emacs. Everyone loved it.
> People still love it. To this very day, I still have to listen to long stories from our non-technical folks about how much they miss Mailman. I'm not shitting you. Last Christmas I was at an Amazon party, some party I have no idea how I got invited to, filled with business people, all of them much prettier and more charming than me and the folks I work with here in the Furnace, the Boiler Room of Amazon. Four young women found out I was in Customer Service, cornered me, and talked for fifteen minutes about how much they missed Mailman and Emacs, and how Arizona (the JSP replacement we'd spent years developing) still just wasn't doing it for them.
> It was truly surreal. I think they may have spiked the eggnog.
AFAIK, mailman was the only thing ever wrapped in this way.
I wrote all the rest of the back-office utilities (at least, the initial versions of them), and I have never come across any indication that they got "wrapped" in anything else (they did, no doubt, evolve and mutate in something utterly different over time).
Yegge's quote is also slightly inaccurate in that mailman, like all early amzn software, was written in C++. Shel and I just chose not use very much of the (rather limited) palette of C++ syntax & semantics.
> There’s this interesting dissonance where all the second year CS students and their professors agree it’s the wrong tool for the job yet the most successful products in the world did it anyway.
> I guess you could interpret that to mean all these people building these products made a bad choice that succeeded despite using Python but I’d interpret it as another instance of Worse is Better. Just like Linus was told monolithic kernels were the wrong tool for the job but we’re all running Linux anyway.
This isn't the correct perspective or take away. The 'tool' for the job when you're talking about building/scaling a website changes over time as the business requirements shift. When you're trying to find market fit, iterating quickly using 'RAD' style tools is what you need to be doing. Once you've found that fit and you need to scale, those tools will need to be replaced by things that are capable of scaling accordingly.
Evaluating this binary right choice / wrong choice only makes sense when qualified with a point in time and or scale.
YouTube video processing uses C++. It also uses Go and Java along with Python.
PInterest makes heavy use of Erlang for scaling.The rate-limiting system for Pinterest’s API and Ads API is written in Elixir and responds faster than its predecessor.
Takeaway: Basically you need to either build your own PythonVM/CPython fork for better Python performance or use another language for the parts that needs to scale or run fast.
I think people look ahead. They see how the app evolves (from A to B) and then claim "X not good". Where as they do not judge the flexibility of X as a tool to move from A to B. Typically they look at B and make claims from that perspective.
Those companies that succeed in python usually have a long path and python was never successfully removed and most likely attempts were made. The PL economics is often stickyness and its not easy to propose absolute measure.
We might be running Linux, yet it has so many virtualization layers on cloud infrastructure, user space stacks to workaround switching to kernel, with microservices for everything, that it is effectively a monolithic kernel being bended into a microkernel one.
Same thing with Python, those business succeed despite Python, and when they grew it was time to port the code into something else, or spend herculean efforts into the next Python JIT.
+1. Languages that are general purpose get used for everything. Perl for the web. Python for builds. Scala transpiled for web.
Portability has many solutions that are good enough - often only bad because they result in second order issues which are themselves solvable with limited pain. Being able to scale software further without having to solve difficult distributed systems problems is of value.
I want a common language I can work with. Right now, Python is the only tool which fits the bill.
A critical thing is Python does numerics very, very well. With machine learning data science, and analytics being what they are, there aren't many alternatives. R, Matlab, and Stata won't do web servers. That's not to mention wonderful integrations with OpenCV, torch, etc.
Python is also competent at dev-ops, with tools like ansible, fabric, and similar.
It does lots of niches well. For example, it talks to hardware. If you've got a quadcopter or some embedded thing, Python is often a go-to.
All of these things need to integrate. A system with Ruby+R+Java will be much worse than one which just uses Python. From there, it's network effects. Python isn't the ideal server language, but it beats a language which _just_ does servers.
As a footnote, Python does package management much better than alternatives.
JavaScript has finally gone from a horrible, no-good, bad language to a somewhat competent one with ES2015, but it has at least another 5-10 years before it can start to compete with Python for numerics or hardware. It's a sane choice if you're front-end heavy, or mobile-heavy. If you're back-end heavy (e.g. an ML system) or hardware-heavy (e.g. something which talks to a dozen cameras), Python often is the only sane choice.
> As a footnote, Python does package management much better than alternatives
No offense meant, but that sounds like the assessment of someone that has only experienced really shitty package management systems. PyPI has had their XMLRPC search interface disabled for months (a year?) now, so you can't even easily figure out what to install from the shell and have to use other tools/a browser to figure it out.
Ultimately, I'm moving towards thinking that most scripting languages actually make for fairly poor systems and admin languages. It used to be the ease of development made all the other problems moot, but there's been large advances in compiled language usability.
For scripting languages you're either going to follow the path or Perl or the the path of Python, and they both have their problems. For Perl, you get amazing stability at the expense of eventually the language dying out because there's not enough new features to keep people interested.
For Python, the new features mean that module writers want to use them, and then they do, and you'll find that the system Python you have can't handle what modules need for things you want to install, and so you're forced to not just have a separate module environment, but fully separate pythons installed on servers so you cane make use of the module ecosystem. For a specific app you're shipping around this is fine, but when maintaining a fleet of servers and trying to provide a consistent environment, this is a big PITA that you don't want to deal with when you've already chosen a major LTS distro to avoid problems like this.
Compiling a scripting language usually doesn't help much either, as that usually results in extremely bloated binaries which have their own packaging and consistency problems.
This is cyclical problem we've had so far. A language is used for admin and system work, the requirements of administrators grate up against the usage needs of people that use the language for other things, and it fails for non-admin work and loses popularity and gets replaced be something more popular (Perl -> Python) or it fails for admin work because it caters to other uses and eventually gets replaced by something more stable (what I think will happen to Python, what I think somewhat happened to bash earlier for slightly different reasons).
I'm not a huge fan of Go, but I can definitely see why people switch to it for systems work. It alleviates a decent chunk of the consistency problems, so it's at least better in that respect.
>No offense meant, but that sounds like the assessment of someone that has only experienced really shitty package management systems. PyPI has had their XMLRPC search interface disabled for months (a year?) now, so you can't even easily figure out what to install from the shell and have to use other tools/a browser to figure it out.
Yes, this is, frankly, an absurd situation for python.
And then there is the fact that I end up depending on third-party solutions to manage dependencies. Python is big-time now; stop the amateur hour crap.
Most languages have numerous third-party solutions for managing dependencies, or only recently added native support. go only recently added modules and was an absolute mess prior to that. javascript has npm, yarn, and about a million others. PHP has compose, but it doesn't cover everything. C/C++ are a mess. Java has gradle, maven, sbt, etc.
> As a footnote, Python does package management much better than alternatives.
If you use it as a scripting language, that might very well be the case (it's at least simpler). When you're building libraries or applications, no, definitely not. It's a huge mess, and every 3 years or so we get another new tool that promises to solve it, but just ends up creating a bigger mess.
This is an overly-dire assessment in my opinion. Setuptools + Pip has been more or less stable for years, and no changes in dev environment have been needed since stability was reached. There is a lot of new stuff coming out, for people who want new stuff, but there's nothing wrong with the old stuff if the old stuff works for you, which will remain supported for several more years if not forever.
Oh, there are a half dozen different tools that solve python package management. Unfortunately, they are mutually incompatible and none solve it for all use cases.
It’s somewhat the opposite situation, poetry is a tool that lots of people are adopting rather than one being pushed by a standards committee. I don’t think it set out to unify a dozen standards, only build a good UX and to be reproducible.
There are good technical reasons to use Fortran even in 2022. For example, it avoids** is aliasing (pointers / references / etc.). This allows for some kinds of optimizations impossible in most other languages.
It's used in a bunch of small niches, but it has users beyond just astronomy.
It's not C code. It calls into a mixture of C, CUDA, Fortran, and a slew of other things. Someone did the work of finding the best library for me, and integrating them.
As for me, I write:
A * B
It multiplies two matrices. C can't do that. In C, I'd have some unreadable matrix64_multiply(a, b). Readability is a big deal. Math should look more-or-less like math. I can handle 2^4, or 2**4, but if you have mpow(2, 4) in the middle of a complex equation, the number of bugs goes way up.
I'd also need to allocate and free memory. Data wrangling is also a disaster in C. Format strings were a really good idea in the seventies, and were a huge step up from BASIC or Python. For 2022?
And for that A * B? If I change data types, things just work. This means I can make large algorithmic changes painlessly.
Oh, and I can develop interactively. ipython and jupyter are great calculators. Once the math is right, I can copy it into my program.
I won't even get started on things like help strings and documentation.
Or closures. Closures and modern functional programming are huge. Even in the days of C and C++, I'd rather do math in a Lisp (usually, Scheme).
I used to do numerics in C++, and in C before that. It's at least a 10x difference in programmer productivity stepping up to Python.
Your comment sounds like someone who has never done numerical stuff before, or at least not serious numerical stuff.
Also, anyone doing any remotely serious number crunching/linear algebra work is well aware that you need to have control over which algorithms you use to run these primitive operations, and which data type you're using.
> I used to do numerics in C++, and in C before that. It's at least a 10x difference in programmer productivity stepping up to Python.
I'm rather skeptical of your claim. Eigen is the de-facto standard C++ linear algebra toolkit and it overloads operators for basic arithmetics.
I'm not sure your appeal to authority is backed up with any relevant experience or authority.
I'm not sure your appeal to authority is backed up with any relevant experience or authority. It's ok if you like Python and numpy, but don't try to pass off your personal taste for anything with technical merit.
I haven't met a single person who did any number crunching work whatsoever who ever experienced any problem doing basic matrix-matrix products with BLAS. Complaining about flags to handle row-major/column-major matrices while boasting about being an authority on number crunching is something that's not credible at all.
I don't actually mind memorizing these things if I'm doing this 60 hours per week. I used to be quite good at some of the C++ STL numerics libraries. I took the same perverse pride some folks here apparently do in knowing these things in-
and-out.
However, most of code is about being able to read it and modify it, not write it. Equations are sometimes hard enough when they look like equations. If you've got a call like that in your code, that's going to be radically harder to understand than A*B.
That's not to mention all the clutter around it, of allocating and deallocating memory.
One of the things C++ programmers don't typically understand are the types of boosts one gets from closures, garbage collection, and functional programming in general, especially for numerics. I recommend this book:
- The system symbolically computes the Lagrange equations (which are derivatives of the above)
- This is compiled into native code
- A numerical algorithm integrates that into a trajectory of motion
- Which is then plotted
All of this code is readable (equations are written in Lisp, but rendered in LaTeX). None of this is overly hard in a Lisp; this was 1-3 people hacking together, and not even central to what they were doing. It'd be neigh-impossible in C or C++.
Those are the sorts of code structures which C/C++ programmers won't even think of, because their impossible to express.
(Footnote: Recent versions of C++ introduced closures; I have not used them. People who have express they're not "real closures.")
That's the shocking thing. It's totally not hard in a Lisp. I could write the code for that symbolic derivation in an evening, tops.
The hard part is the compiler. The somewhat hard part is the efficient numerical integrator (if you want good convergence and rapid integration). The symbolic manipulation is easy, once you know what you're doing. If you want to know what you're doing, it's sufficient to see how other people did it:
If you're dumb like me, and can't write a compiler or an efficient numerical integrator in your spare time, having this interpreted and using a naive integrator is still good enough most of the time. Computers are fast. The authors of the above proved the motion of the solar system is chaotic with a very, very long, very, very precise numeric integration on hardware from decades ago, so they have super-fancy code. For the types of things I do, a dumb integration is fine.
Once you've seen how it's done, and don't mind lower speed, that's trivial in any language with closures (e.g. Python or even JavaScript). If you want to swing a double-pendulum, or play around with the motion of a solar system over shorter durations, it's easy.
And using the tool is even easier. Look at SICM (linked book). The things that look like code snippets are literally all the code you need.
Wow, I can't believe I've never heard of this book (SICM, not the wizard book). Seems incredibly cool.
That said, I'll maintain that my initial skepticism was somewhat justified given that the derivation relies on a pre-written symbolic manipulation/solver library (scmutils) so it's not quite 'from scratch' in a Lisp ;) although I believe that you could write such a library yourself, as SICP demonstrates.
As a side note, it looks like I can't go through this book because I have an M1 Mac, which GNU Scheme doesn't support :(
In case you are forced to use the unreadable long-named unintuitively-syntaxed methods, add unit tests, and check that input-output pairs match with whatever formula you started with.
Yet, Python (and most of her programmers including data scientists, of which I am one) stumble with typing.
if 0.1 + 0.2 == 0.3:
print('Data is handled as expected.')
else:
print('Ruh roh.')
This fails on Python 3.10 because floats are not decimals, even if we really want them to be. So most folks ignore the complexity (due to naivety or convenience) or architect appropriately after seeing weird bugs. But the "Python is easiest and gets it right" notion that I'm often guilty of has some clear edge cases.
Why would you want decimals for numeric computations though? Rationals might be useful for algebraic computations, but that’d be pretty niche. I’d think decimals would only be useful for presentation and maybe accountancy.
If you want that you should use integers. This seems to be a misalignment of expectations rather than a fault in the language.
Other people have posted other examples but it’s not possible to represent real numbers losslessly in finite space. Mathematicians use symbolic computation but that probably is not what you would want for numerics. I could see a language interpreting decimal input as a decimal value and forcing you to convert it to floating point explicitly just to be true to the textual representation of the number, but it would just be annoying to anyone who wants to use the language for real computation and people who don’t understand floating point would probably still complain.
Edit: I’ll admit I have a pet peeve that people aren’t taught in school that decimal notation is a syntactic convenience and not an inherent property of numbers.
That's slightly different in that most programmers won't read 1/3 as "one third" but instead "one divided by three", and interpret that as three divisions added together, and the expectations are different. Seeing a constant written as a decimal invites people to think of them as decimals, rather than the actual internal representation, which is often "the float that most closely represents or approximates that decimal".
It is not a Python thing, it is a floating-point thing. You need it if you want hardware support (CPU/GPU) for non-integer arithmetic in any language. Otherwise, you have decimal, fractions, sympy, etc modules depending on your needs.
This is an issue for accountancy. Many numerical fields have data coming from noisy instruments so being lossy doesn't matter. In the same vein as why GPUs offer f16 typed values.
> That's not Python doing numerical stuff. That's C code, called from Python.
That's sort of a distinction without a difference, isn't it? Python can be good for numeric code in many instances because someone has gone through the effort of implementing wrappers atop C and Fortran code. But I'd rather be using the Python wrappers than C or especially Fortran directly, so it makes at least a little sense to say that Python "does numerics [...] well".
> Not unless they're pushed to, like Python was.
R and Matlab, maybe. A web server in Stata would be a horrible beast to behold. I can't imagine what that would look like. Stata is a terrible general purpose language, excelling only at canned econometrics routines and plotting. I had to write nontrivial Stata code in grad school and it was a painful experience I'd just as soon forget.
You can do web stuff in R, but it's a lot harder than it needs to be. R sucks for string interpolation, and a lot of web related stuff is string interpolation.
Yeah, I'm not surprised by that. The extent of my web experience in R is calling rcurl occasionally, so I've never tried and failed to do anything complicated.
> the problem is using Python at all for a web server
I don't agree with this. Maybe for a web server where performance is really going to matter down to the microsecond, and I've got no other way to scale it. I write server code in both Javascript and Python, and despite all of my efforts I still find that I can spin up a simple site in something like django and then add features to it much more easily than I can with node. It just has less overhead, is simpler, lets me get directly to what I need without having to work too hard. It's not like express is hard per se, but python is such an easy language to work with and it stays out of my way as long as I'm not trying to do exotic things.
And then it pays dividends later, as well, because it's really easy for a python developer to pick up code and maintain it, but for JS it's more dependent on how well the original programmer designed it.
The problem with Django services is the insanely low concurrency level compared to other server frameworks (including node).
Django is single request at a time with no async. The standard fix is gunicorn worker processes, but then you require entire server memory * N memory instead of lightweight thread/request struct * N memory for N requests.
I shudder to think that whenever Django server is doing an HTTP request to a different service or running a DB query, it's just doing nothing while other requests are waiting in the gunicorn queue.
The difference is if you have an endpoint with 2s+ queries taking 2s for one customer, with Django, it might cause the entire service to stall for everybody, whereas with a decent async server framework other fast endpoints can make progress while the 2s ones are slow.
Django has async support for everything except the ORM. async db is possible without the ORM or by doing some thread pool/sync to async wrapping. A PR for that was under review last I checked.
Either way, high concurrency websites shouldn't have queries that take multiple seconds and it's still possible to block async processes in most languages if you mix in a blocking sync operation.
We use the ncpu * 2 + 1 formula for the number of workers that serve API requests.
I don't think in 'handling x concurrent requests' terms because I don't even know what that means. Usually I think around thoughout, latency distributions and number of connections that can be kept open (for servers that deal with web sockets).
For example if you have the 4 core computer and you have 4 workers and your requests take around 50ms each you can get to a throughput of 80 requests per second. If the fraction of request time for IO if 50% you can bump your thread count to try to reach 160 request per second. Note that in this case each request consumes 25ms of CPU so you would never be able to get more than 40 requests per second per CPU whether you are using node or python.
This sounds like sour grapes. Python is a general-purpose language. Languages like Awk and Perl and Bash are clearly domain-specific, but Python is a pretty normal procedural language (with OO bolted on). The fact that it is dynamic and high-level does not mean it is unsuited for applications or the back-end. People use high-level dynamic languages for servers all the time, like Groovy or Ruby or, hell, even Node.js.
What about Python makes it unsuitable for those purposes other than its performance?
Totally agree that performance is not on my top 10 wish list for Python.
But I disagree on "not the right jobs for the tool".
Python is extremely versatile and can be used as a valid tool for a lot of different jobs, as long as it fits the job requirements, performance included.
It doesn't require a CS degree to know that fitting job requirements and other factors like the team expertise, speed, budget, etc, are more important than fitting a theoretical sense of "right jobs for the tool".
> It doesn't require a CS degree to know that fitting job requirements and other factors like the team expertise, speed, budget, etc, are more important than fitting a theoretical sense of "right jobs for the tool".
It requires experience.
A lot of those lessons only come after you've seen how much more expensive it is to maintain a system than to develop one, and how much harder people issues are than technical issues.
A CS degree, or even a junior developer, won't have that.
Whether Python will be easier or harder to maintain depends on numerous factors that vary so much for each job that you cannot generalize upon.
That's something experience shows.
Reaching such a conclusion that "Python is not a right tool for web backend" is just naive.
No matter how experienced a developer is, reality of the world is at least 100x more diverse than what they alone could possibly have learned and experienced.
If one believes to possess the experience of everything to generalize on complex topics like this, it just shows this person could benefit from cultivating a bit more humbleness.
The world doesn't revolve around web development. It's not the only use case. Scientific Python is huge and benefits tremendously from the language being faster. If Python can be 1% faster, that's a significant force multiplier for scientific research and engineering analysis/design (in both academia and industry).
Because most of the really huge scientific Python libraries are written as wrappers over lower-level language code, I'd be curious to what extent speeding up Python by, say, 10% would speed up "normal" scientific Python code on average. 1%? 5%?
I'm not sure it's very relevant to say in a discussion of the answer of "how do we improve Python" is "don't use Python". People have all kinds of valid reasons to use Python. Let's keep this on topic please
I agree! Here's a related point: Rust seems ideal for web servers, since it's fast, and is almost as ergonomic as Python for things you listed as cumbersome in C. So, why do I use Python for web servers instead of Rust? Because of the robust set of tools of tools Django provides. When evaluating a language, fundamentals like syntax and performance are one part. Given web server bottlenecks are I/O limited (mitigating Python's slowness for many web server uses), and that I'd have to reinvent several wheels in Rust, I use Python for current and future web projects.
Another example, with a different take: MicroPython, on embedded. The only good reason I can think for this is to appeal to people who've learned Python, and don't want to learn another language.
>Instead of performance, I’d like to see more effort in portability, package management, and stability for Python because, essentially since it is often enterprise managed, juggling fifteen versions of Python where 3.8.x supports native collection typing annotations but we use 3.7.x, etc. is my biggest complaint. Also up there is pip and just the general mess of dependencies and lack of a lock file. Performance doesn’t even make the list.
I have been leading a Python project lately and, yes, the tooling is very poor, although it is getting better. I have found poetry to be a very good for venv management and lock files + having one file for all your config.
>Python shines beautifully as a (somewhat) cross platform scripting language
Python is much more than just a scripting language. I remember attending this talk[1] a few years about JPMorgan's 35 million LOC Python codebase. Python is being used to built seriously large software nowadays and I don't think performance is ever a minor issue. It should always be in the top 3 for any general purpose language because it directly translates into development speed, time and money.
What compiled languages do you have in mind? I suppose technically there are repls for C or Rust or Java, but I wouldn't consider them ideal for interactive programming. Functional programming might do a bit better -- Scala and GHCi work fine interactively. Does Go have a repl?
Might be tripping you up. Very few languages require that implementations be compiled or interpreted. For most languages, having a compiler or interpreter is an implementation decision.
I can implement Python as an interpreter (CPython) or as a compiler (mypyc). I can implement Scheme as an interpreter (Chicken Scheme's csi) or as a compiler (Chicken Scheme's csc). The list goes on: Standard ML's Poly/ML implementation ships a compiler and an interpreter; OCaml ships a compiler and an interpreter.
There are interpreted versions of Go like https://github.com/traefik/yaegi. And there are native-, AOT-compiled versions of Java like GraalVM's native-image.
For most languages there need be no relationship at all between compiler vs interpreter, static vs dynamic, strict or no typing.
Instead of performance, I’d like to see more effort in portability, package management, and stability for Python because, essentially since it is often enterprise managed, juggling fifteen versions of Python where 3.8.x supports native collection typing annotations but we use 3.7.x, etc. is my biggest complaint. Also up there is pip and just the general mess of dependencies and lack of a lock file. Performance doesn’t even make the list.
This is not to discredit anyone’s work. There is a lot of excellent technical work and research done as discussed in the article. I just think honestly a lot of this effort is wasted on things low on the priority tree of Python.