Python is perfectly fine for glueing together functionality that deals with high latency systems, like a data pipeline that executes queries that are running for minutes. It does not matter from the performance point of view if this code is in Forth or Python, but it matters from the point of view how long does it take to implement and how many engineering hours need to go into it. This is why Python is a good option and F#/OCaml/Clojure is not going to bring much to the table in such cases. Even if I want to implement the ETL pipeline in Clojure or OCaml the rest of the engineers are not ok with these languages, the build systems involved and package management or IDE options, integrations. These are also factors when you select a language. Clojure is my absolute favourite language but there are no people in most of the companies where I work who could buy into it. Management sees it as a risk, not to be able to hire engineers for it. There are many dimensions to this problem than it appears at first.
I would not advertise Python as a language with good IDE support nor package management. I fight with both of these regularly. Also, Python is reasonably suited for gluing together other high performance systems, but not everything in the world is glue code, and as soon as you need to do something O(n) on your dataset, you’re either paying an enormous performance penalty or you’re not writing that bit in Python. People kid themselves into thinking that Python’s C interop will make things fast, and sometimes it does, but it often makes it even slower if the system needs to cross the language boundary O(n) or worse.
> but not everything in the world is glue code, and as soon as you need to do something O(n) on your dataset, you’re either paying an enormous performance penalty or you’re not writing that bit in Python.
And yet Python has one of the richest and most widely used scientific computing stacks. If writing performant code in a friendly language is all that important, then Julia stands as a more reasonable alternative than does some functional language.
>> I would not advertise Python as a language with good IDE support nor package management.
VS Code, VIM works for me. Conda or PIP also. Not sure what is missing for you.
>> but not everything in the world is glue code
I never claimed that.
>> and as soon as you need to do something O(n) on your dataset, you’re either paying an enormous performance penalty or you’re not writing that bit in Python
Depends what you need to do.
My entire comment was about that details matter and you can't just blindly pick a language because of out of the box performance.
Compared with, say, Go where I just hover the cursor.
As for pip, you also need virtual environments to protect you from side effects, and even then, if you’re doing C interop you probably still have dynamic links to so files outside of your virtualenv. Our team spends so much time dealing with environment issues that we’re exploring Docker solutions. And then packaging and distribution of Python artifacts is pretty awful. We’re using pantsbuild.org to build PEX files which works pretty well when it works, but pants itself has been buggy and not well documented.
> I never claimed that
I couldn’t tell since the context of the thread made it sound like you were either implying that Python is suitably performant because the majority of programming is glue code or you were going somewhat off topic to talk about glue code. I now understand it was the latter.
> Depends what you need to do.
My entire comment was about that details matter and you can't just blindly pick a language because of out of the box performance.
I agree, but in practice you rarely know the full extent of what you will need, so you should avoid painting yourself into a corner. It really doesn’t make sense to choose Python any more if you are anything less than certain about the performance requirements for your project for all time—we now have languages that are as easy to use as Python (I would argue even easier, despite my deep familiarity with Python) and which don’t paint you into performance corners. Go is probably the best in class here, but there are probably others too.
Sorry no offence but I do not want to write Go at all. If I want to use such a language I will use Rust with nicer features and better out of the box performance (see TechEmpower results), no GC and more safety (no memory corruption bugs or data races).
I am not sure if I am the one who paints himself into a corner.
Go for it (why would I take offense?). Rust is a fine choice. I'd pick Rust if I ever really needed to eek out every bit of performance out of my system and/or needed safety and was willing to trade quite a lot of developer productivity to get there. Mission critical real-time systems, high-end games, resource-constrained devices, etc are good applications for Rust (and of course there are many others). Rust is a great language and I'd like to use it more often; I just can't justify the productivity hit.
If Rust ever approaches Go's ease of use/learning curve/etc without losing its performance or incurring other costs, I'll happily make the switch for my productivity-sensitive applications as well.
You missed my point—the original comment argued that the software industry is in such great need for performance that functional programming is unacceptable (apart from niche use cases). If that we’re true, then Python and other slow languages wouldn’t be so wildly used. I fully agree that we’re leaving performance on the table by not using functional languages or virtually any other tier of languages including my personal favorite: Go.
