Python has a lot of problems that really slow down development, but they are all fixable.
The biggest issue, in my opinion, is in dependency management. Python has a horrible dependency management system, from top-to-bottom.
Why do I need to make a "virtual environment" to have separate dependencies, and then source it my shell?
Why do I need to manually add version numbers to a file?
Why isn't there any builtin way to automatically define a lock file (currently, most Python projects just don't even specify indirect dependency versions, many Python developers probably don't even realize this is an issue!!!!!)?
Why can't I parallelize dependency installation?
Why isn't there a builtin way to create a redistributable executable with all my dependencies?
Why do I need to have fresh copies of my dependencies, even if they are the same versions, in each virtual environment?
There is so much chaos, I've seen very few projects that actually have reproducible builds. Most people just cross their fingers and hope dependencies don't change, and they just "deal with" the horrible kludge that is a virtual environment.
We need official support for a modern package management system, from the Python org itself. Third party solutions don't cut it, because they just end up being incompatible with each other.
Example: if the Python interpreter knew just a little bit about dependencies, it could pull in the correct version from a global cache - no need to reinstall the same module over and over again, just use the shared copy. Imagine how many CPU cycles would be saved. No more need for special wrapper tools like "tox".
I've always seen it like this: Not everyone builds reproducible software with Python (or in general) and how you handle dependencies can vary. Python leaves it open how you do it: globally installed packages, local packages, or a mix of both.
In the end, it needs to find the import in the PYTHONPATH, so there's no magic involved, and there are multiple robust options to choose from.
So instead of bashing Python for not shoveling down an opinion on you, it's up to the developers to choose which tools they want to use.
If they don't choose one and are unable to freeze their dependencies, it's not a Python problem, but IMO lack of skill and seniority.
You can have both: provide a sane default for most users and allow people to roll their own.
The reason why Python gets extra criticism for this is because it likes to tell people that there should be one obvious way to do it and that it comes with batteries included yet it's dependency management system is just crap and doesn't follow that at all.
Yes :-) It's fair to say Python's approach to dependency management doesn't follow the Zen of Python, but there's a simple way documented in the tutorial:
https://docs.python.org/3/tutorial/venv.html
The fact that there's more than one way to do things in Python is why i've found it so easy and flexible, I have no idea why that goober put this motto in the zen
It's general design guideline and I like Zen of python PEP-20. Explicit is better than implicit and most packaging system in python are explicit which I like. Been using it for over 15 years after perl and been happy with it.
Nothing to complaint as every language has their own set of good and bad. This is what makes it interesting, there is always a room to improve and make things better.
I think they could learn a lot from Rust, which has a very usable, clearly defined way of listing and making dependencies. You can decide how you want to handle individual dependencies (version number, version range, git commit hash, wildcard, etc). I'm not sure how binary dependencies work (i.e. something from your system's package manager), but I've used projects that use them, so the problem is solvable.
Python has always stood out for me as a particularly odd way of doing it. It feels a bit like more like C, but with a package manager that's not quite as nice as other scripting languages have.
It's from the days when Perl was Python's main rival (the late 90s / early 00s). Perl has complex syntax and the"there's more than one way to do it" motto.
Syntactically, and especially in early Python, there were fewer ways of doing things than in Perl and Python people saw that as a positive.
Wait, are you seriously complaining about executing code you downloaded from the internet, that installs a package manager - i.e. a piece of software that downloads executable code from the internet?!
I think what the comment you are replying to are getting at is the fact that installing pip packages from the Internet and importing them in your python app is not that different from piping code from the Internet into your python executable. In both cases python code from the Internet will be executed with your user privileges from within Python. Unless you audit every python package you consume, you might as well accept a curl https://example.com | python installer too.
Yeah, I hate this trend. Unfortunately, you can't pip install poetry because it needs to manage packages, so I guess a different way was necessary. Still, OS-specific packages would be nice, I guess they just need volunteers.
It’s running over HTTPS from an auditable source. Is that _really_ so much worse than a pip install, and can you explain in detail why you believe that to be true?
I teach my kids to use the right tool for the job, because using the wrong tool for the job can lead to injuries. But I violate this all the time, myself. It's just a good habit to get into.
"curl | bash" is a bad habit to get into. It works under certain circumstances, like making sure it's an SSL connection from a source you trust. But it's just a bad habit for the average person to get into.
Yes, funny, but seriously, where's the threat model where you've analyzed the risks of installing code from GitHub over HTTPS and found it to be less secure?
To be clear, either of these methods can have problems, it's not unique to curl and your shell of choice. Some of the better open source projects will say up front that if you are concerned about this kind of thing, feel free to read the installer script and decide for yourself if everything's kosher.
Yes, my point was that if you're worried about running someone else's code the answer is to audit that code rather than the transport layer. There are valid concerns with HTTP or in scenarios where something could be targeted to a single user, but neither of those are relevant to 99% of the time people raise this complaint.
There's always the risk that the script will fail to completely download and leave your system in a broken state. This can be mitigated against by the script authors by wrapping everything in a function which is called on the last line, but how do you know they've done that without downloading the script and checking first?
Do you believe GitHub has that infrastructure deployed? If not, this is a blind alley to worry about. If so, what other precautions have you taken to avoid compromised tarballs, unauthorized pushes to repos with auto-deployment pipelines, etc.?
The point is that in reality you’re orders of magnitude more likely to be compromised by ads in your browser, an undetected flaw in legitimate code, or a compromised maintainer than GitHub having deployed custom infrastructure to target you. If you’re being target by a government, why would they do this instead of using the same TLS exploit to serve you a dodgy Chrome or OS update which is harder to detect and will work against 100% of targets?
So because ads can compromise us we should ignore the security of package managers?
How about this for a reason, where are the checksums when I’m curling and piping? How do I validate in an automated fashion the validity of this file I’m piping into an interpreter? When installing a package it’s quite easy to have redundant copies of an index with checksums pointing to a repository hosting the actual code. The attack surface is much smaller vs a curl | python
This is bad practice, stop promoting it or downplaying it’s security issues.
HTTPS has checksums, and note that we’re specifically talking about installing from Github, where every change is tracked.
> This is bad practice, stop promoting it or downplaying it’s security issues.
I’m trying to get you to do some security analysis focused on threats which are possible in this model but not the real alternatives (download and install, install from a registry like PyPI or NPM, etc.). So far we have “GitHub could choose to destroy their business”, which seems like an acceptable risk and about the same as “NPM could destroy their business”.
HTTPS doesn’t know if the file changed on the server so that doesn’t count here.
I am doing security analysis. If this file changes and I’m using it in built server images then I have no way of automatically validating the changes are good without doing the checksumming myself and managing this data. What we have is a server that can be hacked and the files are unable to be verified by checksum
> Also installable via pip, but... "not recommended", and:
If you install it via pip you need to update it via pip, the alternative would be insane. And the reason it's not recommended is that it doesn't let you use multiple Python versions, but if you're only using one version then installing by pip works fine.
They could just as easily add the same code to setup.py, and then pip would run it as soon as you run pip install. There's generally no security difference between curl | python and pip install.
I agree. Most of the issues the parent mentions have been solved with poetry and pipenv.
And if you need "to create a redistributable executable with all your dependencies". You can either use pyinstaller [0] or nuitka [1] both of which are very actively maintained/developed and continually improving.
Pipenv is plagued with problems and issues. It takes half an hour to install dependencies to our project. The —keep-outdated flag doesn’t (didn’t?) work, so I don’t know if my pipfile is being modified because the constraints require changing versions or because the package manager is errantly updating versions to latest. There are mixed messages about the kind of quality the project aims for. I would not recommend.
Frankly I’ve been burned enough that I won’t use any new packaging technology for Python because everyone thinks they’ve solved it, but once you’re invested you run into issues.
Anyone considering it for production usage should note that package installs in the current versions are much slower than pip or Pipenv. This might affect your CI/CD.
Could you give some details as to why it's better than other more commonly used tools (pip, venv, ...)?
Looking at the home page it's not immediately obvious to me. For example, the lock file it creates seems to be the equivalent of writing `pip freeze` to the requirements file. I see a quick mention of isolation at the end, it seems to use virtual environments, does it make it more seamless? What's the advantage over using virtualenv for example?
I'm not an expert on the internals, but virtualenv interactions feel more seamless. When you run poetry, it activates the virtualenv before it runs whatever you wanted.
So `poetry add` (it's version of pip install) doesn't require you to have the virtualenv active. It will activate it, run the install, and update your dependency specifications in pyproject.toml. You can also do `poetry run` and it will activate the virtualenv before it runs whatever shell command comes after. Or you can do `poetry shell` to run a shell inside the virtualenv.
Python's dependency hell is what made me first look at Julia. I develop on Windows (someone has to :) ), and it was just impossible to get all of the numerical libraries like pydstool, scipy, FEniCS, Daedalus, etc. playing nicely together... so I gave Julia a try. And now the only time I have issues getting a package to run are Julia packages which have a Python dependency. Python is a good language, but having everything in one language and binary-free is just a blessing for getting code to run on someone else's computer.
I've had a good experience with pip-tools (https://github.com/jazzband/pip-tools/) which takes a requirements.in with loosely-pinned dependencies and writes your requirements.txt with the exact versions including transitive dependencies.
Same here, in my team we had immediate dependencies defined in setup.cfg when PR was merged, a pip-compile was run and generated requirements.txt and store it in central database (in our case it was consul because that was easiest to get without involving ops).
pip-sync was then called to install it in given environment, any promotion from devint -> qa -> staging -> prod, was just copying the requirements.txt from environment earlier and calling pip-sync.
Take my upvote. This has helped us a ton. So nice that it resolves dependencies. Only issue we're running into is that we don't use it to manage our dependencies for our internal packages (only using it at the application level). I've been advocating we change so that we simply read in the generated requirements.txt/requirements-dev.txt in setup.py
Late to the party but `pip-tools` also has a flag for its `pip-compile` flag: `--generate-hashes`. It generates SHA256 hashes that `pip install` checks.
"If two of your dependencies are demanding overlapping versions of a library, pip will not necessarily install a version of this library that satisfies both requirements" e.g. https://github.com/pypa/pip/issues/2775
This is what I've always done. Develop using a few dependencies, freeze, continue development with reproducible builds. It has always included the sub-dependencies in the list so, as far as I can tell, this works great for that case...
> nor does pip know what to do if your transitive dependencies conflict with each another
This is true, but because Python exposes all libraries in a single namespace at runtime, there isn't actually anything reasonable to do if they genuinely conflict. You can't have both, say, MarkupSafe 1.1.1 and MarkupSafe 1.1.0 in PYTHONPATH and expect them to be both accessible. There's no way in an import statement to say which one you want.
However, it's notable that pip runs into trouble in cases where transitive dependencies don't genuinely conflict, too. See https://github.com/pypa/pip/issues/988 - this is a bug / acknowledged deficiency, and there is work in progress towards fixing it.
It would change the semantics of the language. You could also write a sys.path hook to interpret the remainder of the file as Ruby and not Python, were pip so inclined....
(Also it's not clear what those changed semantics would be.)
Import system is pluggable, so the semantics are there to be customized. Sure, it could be abused (as many things in Python), but an import hook that checks for a vendored dependency with specific version, seems like a reasonable way to resolve the problem above.
But it changes the semantics of the rest of the language, e.g., if two modules interoperate by passing a type of a third module between themselves, and now there are two copies of that third module, they can't communicate any more.
Getting this right and reliable would be a) a considerable language design project in its own right and b) confusing to users of Python as it is documented, and in particular to people testing their modules locally without pip. It wouldn't be as drastically different a language as Ruby, but it would certainly be a different language.
>Pip freeze does not resolve transitive dependencies
How? Doesn't pip freeze literally list all packages that's installed in the current environment besides basic toolings such as setuptools (and you could even instruct it to list those as well)?
I'm not sure about conflict resolution when but I run a pip freeze it adds TONS of dependencies outside of the 2-3 I had in my app because those were the dependencies of my dependencies.
I think that is what you want. Having all your dependencies, including their dependencies, explicitly specified (including name and version) is what gives you reproducible builds.
Ruby does the same thing with Gemfile.lock. npm does the same thing with package-lock.json.
You can do that with any library. You can issue Django commands by running `python -m django`; that doesn't change the fact that Django is a completely separate project from Python.
It catches way too much. IPython, black and the testing libraries are _not_ a part of my actual dependencies and shouldn't be installed in production. A good UI for a dependency manager at the very least distinguishes between dev and production context, and ideally lets me define custom contexts.
Why do you want to update your dependencies if they work? Isn't the whole point of dependency management to avoid using different versions of dependencies than the ones they have been tested on?
Security fixes, performance enhancements, new features. There are many reasons. But the point is you update in a controlled manner. You don't just push the latest version of everything out on to prod, but you also don't keep pushing the same version that worked a year ago.
saddened to see this poorly constructed comment berating python at the top of this thread. the author seems to have some personal issues with the language given the generally frustrated tone of the comment. the entire comment could have just been 1 line "We need official support for a modern package management system, from the Python org itself." which would be consumed as constructive feedback by all readers with the right context. but somehow the author chooses to "vent" adding unnecessary drama to something that does not get in the way of writing high quality production grade python apps (general purpose, web, ai or otherwise)
there is no language that is devoid of shortcomings - so to any new (<3 yrs exp) python users, please ignore the above comment entirely as it has no bearing on anything practical that you are doing/will do. and all experienced python users know that there are ways to work around the shortcomings listed here and beyond.
> the author seems to have some personal issues with the language given the generally frustrated tone of the comment
What "personal issues" do you think the author has? The frustrated tone comes from the frustrations the author explicitly outlines; unless you think this shouldn't be so, you are turning this into an ad-hom.
> the entire comment could have just been 1 line "We need official support for a modern package management system, from the Python org itself."
Why? because you don't appreciate the detail on why we need such a thing? These issues certainly get in the way of producing production apps; not in the sense that they make it impossible, but they make the process harder and slower than it needs to be.
I actually quite liked that post. I use python maybe once a year or less, and don't enjoy the experience. That post distinguished some of the details which in my rare usage I see simply as a gloopy mess.
It is funny - half of the real desire/need for containers comes back to these sorts of issue with both node and Python. And then they bring in their own different challenges.
I have been programming with node for the last 3 years and I never had any dependency issues with node (at least for 3rd party dependencies). I cannot say that with python that requires using some tool be it docker or virtualenv to isolate them from the already installed ones.
Node's dependency managers npm/yarn just copy the versioned dependencies from their cache folder into the local node_modules folder and remove transitive dependencies duplicates when possible by flattening them into node_modules.
Lucky! I wrote a small internal app in node for my company that relied on an IMAP library. 3 months after launch, someone upgraded the library and my app stopped working. Stack traces were incomprehensible. No “how to upgrade” documentation in sight.
So I spent 2 hours and rewrote it in Java 8 with Maven.
Issues all gone. Node has some work to do before I’ll consider touching it again.
Having such a basic part of a programming language be awful is inexcusable. It's not just that it takes a lot of time; even if it took no extra time, you're still wasting extra space on your computer, risking breakage on external updates, and compromising security because you can't even tell what code you're running.
50% of C++ devs don't use a package manager and 27% rely on a system package manager [1]. You don't hear C++ devs complaining about these issues not because they're happy with the state of dependency management in C++ but because there's a very low rate of adoption for package management systems. That, and the state of dependency management in C++ was so bad for so long that it's viewed as a fact of life.
Also with C and C++ your dependencies compile with your code into a single binary, unless you explicitly opt into using a library, and when you do it becomes a package manager's issue not yours.
That's vastly oversimplifying the problem. "DLL hell" is a term for a reason. The vast amount of effort and complexity Microsoft has put into managing this problem is proof that dependency management for C and C++ is not a solved problem.
We definitely care about it. And I don't know why you think C++ developers can just push issues onto packagers when 50% don't use any kind of package management system. Meaning compiling libraries from source.
If a project you want to depend on isn't using a dependency management framework, how would you then make it work in your project? You will have to do extra work to define the transitive dependencies!
What needs to happen is standardization - this has been done in java because of it's maturity. There's almost no java project that isn't using the standard maven dependency management (even projects that don't use maven, such as gradle projects, would use maven dependency management, and export themselves as an artifact usable via maven).
Javascript has an even worse problem, so python isn't alone me thinks...
First of all, Poetry locks every dependency (even transitive ones) to the version you know works. This solves the problem of the project not using dependency management properly.
Secondly, setup.py allows you to specify your dependencies there, so most libraries use and specify that, which means that that isn't that much of a problem in Python. Sure, sometimes it is, but I haven't run into that particular problem very often.
Last time I gave it a go, I found it was pretty strongly welded to virtualenv, rather than using Python's own (much less problematic) venv. I came away less than enthused as a result.
(To be fair, modifying it to use venv is... non-trivial).
Pipenv has a good approach to management, similar to npm, but the implementation is buggy. It gained popularity by being recommended very early on in the official python documentation while being misleadingly advertised as production-ready.
If you look at the issues section in its github repo, you'll see that there are some pretty basic bugs there which are very annoying or disruptive. Moreover it seems the author has almost left the boat and a handful of contributors have to tidy things up.
Just to illustrate my point. I think a package manager that takes a few minutes to install a single tiny package, or don't prevent you from adding non-existing packages (e.g. spelling mistake), or doesn't let you install a new package without trying to upgrade all other packages isn't really production-ready. These are known issues since November last year.
Excuse me, but as a long time Python user I have to disagree. I started using Rust two years ago and Rust’s dependency managment is easily the best thing I ever saw (keep in mind that I didn’t see everything, so there is a chance there are better things out there).
The project-/dependency-manager Cargo¹ is more “pythonic” than anything Python ever came up with and where others mumbled ”dependency hell is everywhere” the Rust people seem to have thought something like: ”there must be a way to do this properly”.
The whole thing gives me hope for Python, but angers me everytime I start a new python project. Poetry is good, but there should be an official way to do this.
It saddens me to see, that some people seem to just have given up on dependency managment alltogether and declared it unsolvable (which it is not, and probably never has been).
There's a pattern to this. The later the dependency manager was created, the better it is. This is a hard problem space where each new language got to use the lessons learned on the earlier ones.
Cargo, though, has a silver bullet. If it can't find a solution to determine a single version for a package, it simply includes more than one version in the object code. That would take a lot of work to duplicate in Python.
Unfortunately, pip came along and took over, despite lacking support for that. It would have been nice if a better packaging tool had replaced easy_install, but alas.
They definitly gave this a thought while designing the library system (”crates”) for the language. I am not sure if it is feasible to retrofit such a solution to something like python.. Python 4 maybe?
Rust's cargo, JS's yarn and the grand daddy of them all Ruby's bundler address all these issues. Even newer versions of Gradle support a workflow where you specify the versions you know you want and just on everything else, including transitive dependencies, down.
This is the other frustrating thing: there is this stockholm syndrome effect, because people are so used to dependency management being horrible, they think there are just no good dependency management systems, and they give up.
GitHub is still the host for many of them, but there are Modules, so you get proper versioning and all that even when the place you end up getting them from is GitHub.
I think ruby is alive and well for a lot of startups. I do think it is being squeezed on three sides though.
* From javascript. If you have a app like front end, you are going to use js. Why not have the whole stack be js and have your developers use only one language.
* From python for anything web + data science. Again, why not have your whole stack be in one language?
* From lack of hype. Rails is still evolving, but a lot of packages are not seeing releases (I have used packages 3-4 years old). This indicates to me that the energy isn't there in the community the way it used to be. I have seen the consultants who are always on the bleeding edge move on to elixir.
That said I have seen plenty of startups using ruby (really rails) and staffing when I hired for ruby wasn't an issue.
I do help run a local ruby meetup and attendance is good but not exceptional (15-40 people every month). So that may skew my viewpoint.
"From JavaScript" also includes another side: When your frontend is in JS, your backend can be a simple REST API. And building a REST API requires much less framework than building a server-side-rendering webapp does, so it's tempting to use Go or Rust or whatever you like.
You’ll need (probably) at least:
-Database connection
-An ORM
-Middleware against attacks / rate limiting
-Caching
-Jobs / workers
-A rendering engine for email and maybe pdf
-Some sort of admin/backend
-Logging
-Validation
I’ve written an API once from scratch. Actually twice. First time in Modena, because it was all the hype, but it was arcane. Then Sinatra, where I ended up creating all of the above. Rails is excellent for APIs.
Rust is nice, but I’m not sure if I’d like it for all of an API. I don’t like go. Crystal seems great, because it’s typed and it’s also super fast.
Agreed. I still think that ruby is great for jamming out an API (far better in terms of development speed than go or rust) but a lot of the great gems that can speed up development assume server side rendering. That plus the fact that go/rust/whatever are probably more "interesting" and faster (at runtime) than ruby is an additional obstacle (for ruby!).
I loved Ruby, but unfortunately it didn't hold on to any kind of "killer app" role after Rails clones showed up in other language. I've switched to Elixir/Phoenix in that space and not looked back.
With the rise in ML and data science over the past years, Python finally has a killer app that no other scripting languages come close to touching. I migrated completely from Ruby when I started dabbling in ML, Pandas, etc.
So, as someone who spends maybe 20% of their time hiring, it's still a very effective screen. You wouldn't believe how many people can't do it. People at big companies, respected places. It's surprising.
I used a different screen (having people make change based on an arbitrary amount, so if the input was 81, you'd return [25, 25, 25, 5, 1], as we were in the USA) and it was also helpful. I didn't track the number of people that it stymied though.
Yah, that's also a good one. I like the variant that asks how many different ways you can make change for a given amount and a given array of currencies.
(I always feel weird talking about interview questions publicly, but honestly anyone who prepares that diligently deserves to go to the next stage. If anyone's reading this because they're preparing for an interview with me and I ask this question, just mention this comment and I'll be impressed.)
I am trying to find a place in the industry - again, starting from RoR. I absolutely love Ruby. And all this talk of "Ruby dying" makes me feel sad. The rational thing to do is to move on, and learn something popular, like node.js but the more I see Ruby in action, I just can't pull myself away from it.
I had managed to get a job as a Java developer a long time ago, but at that time all I could do was barely write toy apps in Java and I had no exposure to stuff like design patterns. The whole experience left me in a bad place. Now after all these years, Ruby feels like a breath of fresh air, and the texts that I have come across on the subject - Design Patterns in Ruby, Practical OOD in Ruby, Ruby under a Microscope etc. have increased my interest in the language.
But more and more frequent articles on Ruby's decline are pretty disheartening.
Ruby's future may actually not be Ruby itself. Probably the major problem with Ruby is its performance, which is slow even compared to other interpreted languages. While I'm not sure it is really production ready yet, Crystal is very interesting -- it's a native compiled statically typed language that nevertheless feels very much like Ruby. Check it out if you haven't.
I saw Tenderlove's interview on SE Radio, about Ruby internals. He seemed optimistic, but also because he's been working on a performance related project for the past few years now. Anyway, I'm hopeful.
Python is uniquely ill-suited for dependency management compared to many other languages. For some reason dependencies are installed into the interpreter itself (I know what I just said is very imprecise/inaccurate but I think it gets the point across).
In JS, which also has a single interpreter installed across the system (or multiple if you use nvm), the packages aren't installed "directly" into the interpreter, which removes the need for things like virtual-envs, thus making life a lot easier. I wish Python did something like this.
That being said, pipenv is making things easier. However, I think pipenv is a workaround more fundamental problems.
Thanks for that! I am tired of messing with conda, virtualenv, etc. and since I use simple Makefiles to build and run most of my code, I can easily stick with the standard latest stable version Python installation when using your trick.
EDIT: a question: when I have to use Python, I like to break up my code into small libraries and make wheel files for my own use. How do you handle your own libraries? Do you have one special local directory that you build wheel files to and then reference that library in your requirements.txt files?
> How do you handle your own libraries? Do you have one special local directory that you build wheel files to and then reference that library in your requirements.txt files?
We didn't build wheels. We had a centralized git host (Gitlab, but any of them works) with all our libraries, and just added the git url (git+https://...) to the requirements.txt
It's not a "model," but if you're able to 1) use fewer dependencies 2) use stable dependencies 3) use dependencies with fewer dependencies, it helps with dependency hell. I've even made commits to projects to reduce their dependency count.
I have found that I would rather code my own versions of some libraries so I have control over it. Even if there is some extra long term maintenance and some up front dev costs, it's paid off already a number of times.
A little off topic, but this is why I really like a Common Lisp with Quicklisp: library dependencies are stored in a convenient location locally and the libraries I write can be treated the same way (with a trivial config change to add Quicklisp load paths to my own library project directories).
Indeed. We have some Python modules written in Rust. It needs Rust nightly, because pyo3 requires Rust nightly. The Rust crate relies on libtensorflow. Unit tests for the Python module use Python and pytest. And we use our own build of libtensorflow (optimizations for AVX and FMA).
The dependencies of such projects are easy to specify in Nix. Moreover, it's easy to reproduce the environment across machines by pinning nixpkgs to a specific version.
I installed Nix operating system on an old lap top earlier this year, and indeed it does solve a lot of development and devops problems. I retired this spring, so I only played with Nix out of curiosity, but if I still had an active career as a developer I would use Nix.
I agree that builtin tools suck for dependency management.
However a lot of the issues that you mentioned (such as lock file and transitive dependencies) can be handled by pipenv, which should be the default package manager
Python was a scripting language. All those problems are caused by people using it like something it isn't. Python has way outlived it's usefulness and it's about time we move on to something better.
The virtualenv thing just galls me. Sure, pipenv aped rbenv - appropriately, I might add - but until they supplant virtualenv as the recommended way to have separate environments, I'll pass.
.NETs solution to this was the project file, a configuration file that lists the compiler version, framework version, and dependencies (now including NuGet packages and their versions).
> The biggest issue, in my opinion, is in dependency management. Python has a horrible dependency management system, from top-to-bottom.
I agree, although a lot of it has to do that there's so much misinformation about the web, and many articles recommending bad solutions. This is because python went through many packaging solutions. IMO the setuptools one is the one that's most common and available by default. It has a weakness though, it started with people writing setup.py file and defining all parameters there. Because setup.py is actually a python program it encourages you to write it as a program and that creates issues, setuptools though for a wile had a declarative way to declare packages using setup.cfg file, you should use that and your setup.py should contain nothing more than a call to setup().
> Why do I need to make a "virtual environment" to have separate dependencies, and then source it my shell?
Because chances are that your application A uses different versions than application B. Yes this could be solved by allowing python to keep multiple versions of the same packages, but if virtualenv is bothering you you would like to count on system package manager to keep care of that, and rpm, deb don't offer this functionality by default. So you would once again have to use some kind of virtualenv like environment that's disconnected from the system packages.
> Why do I need to manually add version numbers to a file?
You don't have to, this is one of the things that there's a lot of misinformation about how to package application. You should create setup.py/cfg and declare your immediate dependencies, then you can optionally provide version _ranges_ that are acceptable.
I highly recommend to install pip-tools and use pip-compile to generate requirements.txt, that file then works like a lock file and it is essentially picking the latest versions within restrictions in setup.cfg
> Why isn't there any builtin way to automatically define a lock file (currently, most Python projects just don't even specify indirect dependency versions, many Python developers probably don't even realize this is an issue!!!!!)?
Because Python is old (it's older than Java) it wasn't a thing in the past.
> Why can't I parallelize dependency installation?
Not sure I understand this one. yum, apt-get etc don't parallelize either because it's prone to errors? TBH I never though of this as an issue, because python packages are relatively small and it installs quickly. The longest part was always downloading dependencies, but caching solves that.
> Why isn't there a builtin way to create a redistributable executable with all my dependencies?
Some people are claiming that python has a kitchen sink and that made it more complex, you're claiming it should have even more things built in, I don't see a problem, there are several solutions to package it as an executable. Also it is a difficult problem to solve, because Python also works on almost all platforms including Windows and OS X.
> Why do I need to have fresh copies of my dependencies, even if they are the same versions, in each virtual environment?
You don't you can install your dependencies in system directory and configure virtualenv to see these packages as well, I prefer though to have it completly isolated from the system.
> There is so much chaos, I've seen very few projects that actually have reproducible builds. Most people just cross their fingers and hope dependencies don't change, and they just "deal with" the horrible kludge that is a virtual environment.
Not sure what to say, it works predictable to me and I actually really like virtualenv
> We need official support for a modern package management system, from the Python org itself. Third party solutions don't cut it, because they just end up being incompatible with each other.
setuptools with declarative setup.cfg is IMO very close there.
> Example: if the Python interpreter knew just a little bit about dependencies, it could pull in the correct version from a global cache - no need to reinstall the same module over and over again, just use the shared copy. Imagine how many CPU cycles would be saved. No more need for special wrapper tools like "tox".
There is a global cache already and pip utilizes it even withing an virtualenv. I actually never needed to use tox myself. I think most of your problems is that there are a lot of bad information about how to package a python app. Sadly even the page from PPA belongs there.
It's not that bad if you use the right tools. The two main options are an all-in-one solution like poetry or pipenv, and an ensemble of tools like pyenv, virtualenvwrapper, versioneer and pip-tools. I prefer the latter because it feels more like the Unix way.
Why should Python have some "official" method to do this? Flexibility is a strength, not a weakness. Nobody ever suggests that C should have some official package manager. Instead the developers build a build system for their project. After a while every project seems to get its own unique requirements so trying to use a cookie-cutter system seems pointless.
They almost rescinded that in PEP 357 and ultimately did so in PEPs 468/469. PEP -23 updated the standard library to match, but not until 3.9.1.a. Until then, beware the various blog posts you'll find on Google talking about this concept on 1.x.
It was always an ideal to aim for rather than a strict rule. I don't see any of those PEPs changing the balance enough to claim the principle was dead (maybe a bit injured...)
In general, leaving such things open leads to a proliferation of different 'solutions', as multiple people try to solve the issue... leading to the additional confusion and cognitive load of trying to find a single solution which suits your use-case and works, when often none of them are perfect.
Sometimes a 'benign dictator' single approach has benefits...
The virtual env is really the thing that has stopped me from using python. It's a lovely language but the tooling around it needs a lot of help. I'm sure it will get there though. I mean if the js folks can do it, certainly python can.
If it is running on its own computer, for shell scripting.
If it is trying to process ML data, or running in some cloud provider, or deployed in some IoT device supposed to run for years without maintenance, then maybe yes.
Right, but when you're at that point in performance considerations you already have a team of specialists working on multiple angles in performance.
And precisely, for ML code all python libraries run extremely optimized natively compiled code. The language overhead is a minimal consideration. And for business domain code language performance is rarely the limiting factor.
If your team size is 1 then you're not doing yourself any favor thinking about performance beyond basic usability when dev productivity is a far higher priority.
Thanks, that definitely looks like useful data as a starting point.
1. What is the impact of a continuous long-running process? That is, if instead of trying to calculate a result and then shut down, I'm running a web server 24/7, what's the impact of an interpreted language over a compiled language? (Assume requests are few and I'm happy with performance with either.) This not models web servers but things like data science workloads where one wants to conduct as much research as possible, so a faster language will just encourage a researcher to submit more jobs.
2. According to https://www.epa.gov/energy/greenhouse-gases-equivalencies-ca... , 1 megawatt-hour of fossil fuels is 1559 pounds of carbon dioxide. The site you link calculates an excess of 2245 joules for running their test programs, which is approximately .001 pounds of carbon dioxide, or roughly what a human exhales in half a minute. (Put another way, if using the interpreted language saved even one minute of developer time, it was a net win for the carbon emissions of the program.)
> What is the impact of a continuous long-running process?
OK so you're asking about steady-state electricity consumption of a process that's idling? I would bet that it's still lower for a more energy-efficient language, but let's say purely for the sake of argument that they're both at parity, let's say (e.g.) 0. Now what happens when they both do one unit of work, e.g. one data science job? Suppose you're comparing C and Python. C is indexed at 1 by Table 4, and Python at 75.88. So even ignoring runtime, the Python version is 75 times more power-hungry than the baseline C. And this is for any given job.
> a faster language will just encourage a researcher to submit more jobs.
Sure, that's a behavioural issue. It's not a technical issue so I can't give you a technical solution to that one. Wider roads will lead to more traffic over time. What people will need to realize is that if they're doing science, shooting jobs at the server and 'seeing what sticks' is not a great way to do it. Ideally they should put in place processes that require an experimental design–hypothesis, test criteria, acceptance/rejection level, etc.–to be able to run these kinds of jobs.
> if using the interpreted language saved even one minute of developer time, it was a net win for the carbon emissions of the program
I don't understand, what does a developer's time/carbon emission have to do with the runtime energy efficiency of a program? They are two different things.
> What people will need to realize is that if they're doing science, shooting jobs at the server and 'seeing what sticks' is not a great way to do it. Ideally they should put in place processes that require an experimental design–hypothesis, test criteria, acceptance/rejection level, etc.–to be able to run these kinds of jobs.
Sure, but they don't, and perhaps that's a much bigger issue than interpreted vs. compiled languages - either for research workloads or for commercial workloads. People start startups all the time that end up failing, traveling to attract investors, flying people out to interview them, keeping the lights on all night, heading to an air-conditioned home and getting some sleep as the sun is rising, etc. instead of working quietly at a 40-hour-a-week job. What's the emissions cost of that?
> I don't understand, what does a developer's time/carbon emission have to do with the runtime energy efficiency of a program? They are two different things.
This matters most obviously for research workloads. If the goal of your project is "Figure out whether this protein works in this way" or "Find the correlation between these two stocks" or "See which demographic responded to our ads most often," then the cost of that project (in any sense - time, money, energy emissions) is both the cost of developing the program you're going to run and actually running it. This is probably most obvious with time: it is absolutely not worth switching from an O(n^2) algorithm to an O(n) one if that shaves two hours off the execution time and it takes you three hours to write the better algorithm (assuming the code doesn't get reused, of course, but in many real-world scenarios, the better algorithm takes days or weeks and it shaves seconds or minutes off the execution time). Development time and runtime are two different things - for instance, you can't measure development time in big-O notation in a sensible way - but they're definitely both time.
Correct, and computers continue running. I'm referring to the carbon emissions of the development project itself. The faster the development is done, the sooner you can get on with developing other things.
It's a valid objection to the statement they replied to. Saving developer time does not equate to lower emissions, so it is incorrect to call it a "net win".
Sure and trains burn fuel even when you aren’t using them. But if we look at you carbon footprint it doesn’t seem wise to factor in every single train on the planet in your specific account har because they don’t all air still when you aren’t using them.
When talking about the footprint of a company or a project, then you need to restrict the calculations to the resources they actually use. So if a project uses tools to get a product out quicker that means they’ve spend less human-hours, which have a co2 cost associated with them. Then you can weigh the cost of that tool versus the Human Resources both in a financial sense but also with respect to emissions.
Remember NOT to jump into Python for your new product if don't know Python. If you are developing for a young startup, have time crunch, then stick to what you know.
IF you do not have a language, or know Python a bit, then pick Python. Here are some of the reasons why I stick to Python (young startup/web APIs):
- OOPs is not too strict (might give a headache to some folks)
- Mixins, lambda, decorators, comprehensions - Pythonic ways make me feel productive easily
- Create a data Model, drop into a shell, import and try things
- Can do that on a live server
- Do the same with Controllers, or anything else actually
- really nothing fancy needed
- Command line processing, SSH, OS integration, etc. has so many great libs
- Python Dict someone looks like JSON (this is purely accidental but useful)
- Debugger support even in free IDE like PyCharm Community Ed is great
- Integration to a world of services is so easy, even ones you do not commonly see
- Documentation - many libs have consistent structure and that helps a LOT
- Really large community, perhaps only smaller than Java
- The even larger group of people using Python in all sorts of domains from Biotech to OS scripts
What I would like improved in the language would be an even longer list. Every language has a list like that, but when you are focused on being scrappy and building a product, yet making sure that software quality does not take a big hit, Python helps a lot.
> Remember NOT to jump into Python for your new product if don't know Python. If you are developing for a young startup, have time crunch, then stick to what you know.
Are you saying this as a general maxim (don't try to learn a new tech under pressure) or because of characteristics specific to Python, that make it worse in such a situation than any other language/ecosystem?
I know it's only a one off anecdote but two developers in my team who previously didn't know python used it for a small time-critical project and it was a brilliant success.
The main point I am trying to make is that not all of us are language experts. I absolutely know the vital role that language experts play and that we need to solve the issues. But companies need to build in the meanwhile. Python is easier to learn, this is why it is such a popular language in Universities.
I am not a high IQ person to even grip some of the nitty-gritty underpinnings of a language. But should that stop me from building a product?
In the end, I need a language that is easy to pick-up, be productive, has its heart in the right place. A young Python programmer can web scrap easily or plug into Ansible or so many other things. If you know of another language that would make more practical sense and still be easy to pick-up, I would switch.
Reading this got me thinking and I wonder if other people feel like me about this, so I'm going to share it. This is not serious, but not entirely unserious...
I try to be a good sport about it, but every time I write python I want to quit software engineering. It makes me angry how little it values my time. It does little for my soured disposition that folks then vehemently lecture me about the hours saved by future barely-trained developers who will ostensibly have to come and work with my code. Every moment working with python (and that infernal pep-8 linter insisting 80 characters is a sensible standard in 2019) increases my burnout by 100x.
I try to remind myself that we're trying to make the industry less exclusive and more welcoming to new developers and "old" isn't necessarily "good" (in fact, probably the opposite), but damn I just don't understand it.
It used to be that I could focus on other languages (Erlang, Nemerle, F#, Haskell, Ocaml, even C++) and sort of balm myself. But now, I can't even overcome the sinking feeling as I read the Julia statistics book that I'm going to be dragged back to Python kicking and screaming in the morning, so why even bother?
And frustratingly: it's one of the few languages with decent linear algebra libraries. And that means it's one of the few languages with good ML and statistics support. So it's very hard not to use it because when you want to cobble together something like a Bayesian model things like PyMC or Edward actually give you performance that's obnoxiously difficult to reproduce.
This is what the industry wants and evidently a lot of people are okay with it, but to me it's misery and I can't work out why people seem to like it so much.
I am about to hit a decade of python experience. I work with it daily, all my major codebases are written in it.
I hate it.
It has an ok object system which is possibly its only redeeming quality. I found Racket about 4 years ago, and any new project that I work on will be in Racket or CL.
I could go on at length about the happy path mentality of python and the apologists who are too ignorant of other people's use cases to acknowledge that maybe there might be some shortcomings.
The syntactic affordances are awful and you can footgun yourself in stupidily simple ways when refactoring code between scopes. Python isn't really one language it is more like 4, one for each scope and type of assignment or argument passing, and all the sytactic sugar just makes it worse.
Not to mention that packaging is still braindead. I actually learned the Gentoo ebuild system because pip was so unspeakably awful the moment you stepped away from the happy path.
Blub blub blub, someone help I'm drowning in mediocrity.
I am about to hit two decades, with lot of pauses. I now use it professionally, but it was still my favorite language a decade ago (then I fell in love with Common Lisp and lately Haskell).
I think you need to look at it historically. Against other languages circa 2000, Python was a huge win. Easy to write (compared to C), consistent (compared to PHP), minimalist (compared to Perl), multi-paradigm and playing well with C (compared to Java), with a great standard library (compared to Lisp).
Today, the landscape is very different. Lot of languages took hints from and got influenced by Python (including Racket I am sure). Functional languages have taken off. Type inference is a standard feature.
History is always changing. There is some potential for a next Python, but we don't know what it will look like yet. I suspect it will be functional, but I don't think it will be in the Lisp family. It probably won't happen until after Rust and/or Julia get lot more adopted. Anyway, just like C, Python will be with us for decades to come, for better or worse.
Two decade Perl programmer who's been using a bunch of Python recently. I don't hate it, but I've not been getting any particularly complicated use-cases. I do miss Moose a lot, and I miss Bread::Board and I miss DBIx::Class, which SQLAlchemy does not make up for. The weird scoping didn't take too long to get right, although string interpolation still takes me too much thinking about.
What I am missing from Perl is library documentation and MetaCPAN. It seems that the go-to solution is that you're meant to create some kind of damn mini-site for any even vaguely complicated project you do using Sphinx, which seems bizarro land. Also the _requests_ documentation looks like it was written by Wes Anderson and wtf I hate it. Also I hate that all libraries have cute rather than descriptive names; yes there's some of that in Perl, but it feels like less. Bah humbug. Other than that it's fine.
This resonates with me. I miss working with Perl greatly (my current employer forbids me to write anything on it), and having to deal with Python instead makes me miss it even more.
That doesn’t even work in all situations. What if the system requires development packages? What if it’s a different OS or architecture? Packaging is a nightmare.
That's why Docker is a good idea for packaging things up. You basically get exactly the combination of binaries, libraries, etc. you intend to run. Probably not a bad idea to use it for development as well. Or at least something like pyenv or whatever it is called these days.
They aren't too bad IMO. However, you need to set up a build environment/VM with your choice of your earliest-supported Linux+glibc. Build on old, run on new works well.
Linux backwards-compatibility is pretty good, in that a static binary should run just fine on newer systems. I've had far worse experiences with OpenBSD, where a build on an older version of the OS would never seem to run on a newer system.
Thanks for sharing, it certainly makes me feel better about my troubles with accepting python. And "the happy path" might be a state you spend a /minority/ of your time in, depending on project. So first you spend 10% of your time to fix the 80% of easy features, but the tricky stuff can be more awful than it should be.
And packages. I thought I was just stupid, but thankfully there was more to it.
Quite welcome. I think you are on to something. The thought that comes to mind is "Python is a great language if you want to do exactly what everyone else is doing." Most of the love has nothing to do with the language itself, but rather the fact that people can make use of some library that a big engineering effort produced in another language entirely. Thus they conflate Python with access to that functionality. All of my code bases are pure python (in part because of the packaging issues), so maybe that is the difference, people aren't really Python users, they are numpy, pandas, django, sqlalchemy, or requests users. It would be interesting to compare the view of library maintainers vs just 'users.'
Agreed this is it, when people think Python, they think all the default use cases covered by the many libraries that let you "get things done" quickly with syntactic sugar. Alas, there's more to our work than the first few miles of getting things done. There's that scaling part too and python loses it there: no good threading, no enforced typing, bizarre scoping, half baked module system,etc
> Is it the language, or someone forcing you to use that linter and settings?
It's definitely both.
Preface in true internet style: these are just opinions and you may not share them. That's fine.
I really don't like Python as a language. I don't like its total lack of composability. I don't like its over-reliance on a very dated vision of OO. I don't like how its list comprehensions aren't generic. I don't like how it farms all its data structures out to C. I don't like how it uses cruddy one line lambdas and forces me to name things that are already named by the things I'm passing it to.
And also the linter just exacerbates these things, because the linter is just a crystallized slice of a kind of software engineering culture I really don't like.
Not sure why Python’s list comprehensions not being generic might be such a great downside but just wanted to add that I personally think that they are perfectly fine just the way they are. For the context, I’ve been professionally writing Python code for 14 years now. I understand it’s not the perfect language but for quite average programmers like myself it does an excellent job of getting out of the way and of letting you focus just on the data, because that’s what programming is (or should be), i.e. managing and processing data.
Like I said, it’s not the perfect language, it does not have Erlang’s pattern matching nor Lisp’s insistence on thinking about almost everything in a recursive manner, but you can at least mentally try to incorporate those two philosophies into Python once you know about them (and once you know about other similar such great programming concepts).
> I'm not really sure what your point is in criticizing "farming" to C.
If Python is so great how come it can't even express a decent data structure that it needs? I'll happily level the same criticism at Ruby if you like.
> You made a false claim.
Firstly: This is not High School Debate Club. There isn't some kind of point system. Language and expectations like this are not only counterproductive (in that they essentially turn every conversation into a series of verbal ripostes in which the goal is to be most right rather than *learn the most) they're also tedious and unwelcome.
> You referred to Python iteration as not generic,
I did not. I said list comprehensions weren't generic, and then I tried to explain my complaint. It may be that someone has done a legendary feat of going through and creating a bunch of mixins for some of the gaps in the list comprehension coverage such that you can find some way to override how non-determinism is expressed. If so, please point me to it.
> If Python is so great how come it can't even express a decent data structure that it needs?
Why is this a requirement of a "great" language? By not requiring a language to be self-hosting, you are adding more degrees of freedom to your language's design, so I could even see an argument that writing it in C is an improvement. I don't necessarily agree with that, but I don't see why cpython written in C implies it is a bad language. Maybe you could elucidate your thinking?
you are adding more degrees of freedom to your language's design
Why do you think that? Languages/runtimes with high C integration and fairly exposed C bowels like Python and Ruby have, over time, turned out to be very hard to evolve compatibly.
Because it's a fact? With cpython you can develop things in python if that suits you or C if that suits you. You have more freedom to choose what fits your use case. I'm not saying this is necessarily good, but I don't think it's obviously bad. I'd like to hear from people who think it's the case.
> Languages/runtimes with high C integration and fairly exposed C bowels like Python and Ruby have, over time, turned out to be very hard to evolve compatibly.
That is true, but it's also arguably one of the reasons cpython became so popular in the first place. The ability to write C-extensions when appropriate has been very powerful. It's certainly caused issues, but I think if python didn't have exposed bowels it may never have become nearly as popular. What if numpy wasn't ever written? (This isn't to say that they couldn't have exposed a better C-api with fewer issues, but hindsight is 20/20...)
I guess I don't understand how 'the design gets stuck in amber' (which you seem to agree with) and 'gives you lots of design degrees of freedom' can be true at the same time.
It gives you flexibility in writing libraries while making it harder to design a new compatible runtime. That said, PyPy has achieved pretty good C extension support while making the language faster.
The claim was 'degrees of freedom in your language's design'. It's an odd one because the history of a bunch of similar languages has been exactly the opposite. Compare, say, JS to Ruby and Python. Even with the seemingly crippling burden of browser compatibility, Javascript has evolved out of its various design and implementation ruts a lot more gracefully than either Ruby or Python.
> Language and expectations like this are not only counterproductive ... they're also tedious and unwelcome.
Who gets to be the language police? I'm fine with "High School Debate" but "verbal ripostes in which the goal is to be most right rather than learn the most" is not an accurate description of that.
You can do this if you like. In the past, I'm guilty of it as well.
But I won't engage with someone who does this the same way, because they're not engaging me as a human. Unless, of course, they're already dehumanizing me (as occasionally happens on this website) and then I don't feel quite so bad about it.
If Python is so great how come it can't even express a decent data structure that it needs?
I'm sure you know this is a deliberate design choice/tradeoff. It's arguably turned out to be a bit of a millstone for languages that eventually want to grow up to be general-purpose-ish, but that wasn't as obvious at the time.
> You made a false claim. You referred to Python iteration as not generic, when what you really meant is that Python lacks first class monad support.
Are you really that lacking in self-awareness that you responded to someone annoyed about programming culture by going full "debate with logic, facts and reason" mode?
Just a point about linters. They are bad in every language (because everyone has different opinions on what they consider beautiful code).
If you're having a burnout because the PR you just opened are being rejected by the linter you probably just make the linter apply the modifications automatically in pipeline.
We used to hate the linter checks in my current workplace because it was really boring to fix the issues. Now the CI simple fix them, and nobody cares anymore.
I'm not the person you're responding to but what they say resonates with me as a Python developer.
I believe Python is quite possibly the best language for a few things
* Exploratory programming - such as what data scientists do
* Writing programs that will never grow beyond 150LOC, or roughly what fits on a screen + one page down
When I have those two constraints met I am almost always choosing Python.
Here are some problems I face on codebases as they scale up:
* Python conventions lead to what I consider bad code. Devs will often choose things like 'patch' over dependency injection, and I have seen on multiple occasions projects derided for providing DI based interfaces - "oh, why would you write code that looks like Java? This is Python".
There's a lot of death by a thousand cuts. Keyword args in functions are abused a lot, because they're "easy", ability to patch at runtime means it's often "easy" to just write what looks like a simple interface, but it's then harder to test. Inheritance is "easy" and often leads to complex code where behaviors are very, very non-local.
Dynamic types mean that people are often a little too clever with their data. I've seen APIs that return effectively `Union[NoneType, Item, List[Item]]` for absolutely no semantic reason, without type annotations, meaning that if you assumed a list or none came back you were missing the single Item case. The implementation actually internally always got a list back from the underlying query but decided to special case the single item case... why? I see this sort of thing a bit, and other languages punish you for it (by forcing you to express the more complex type).
* I find myself more and more leveraging threads these days. I did this often with C++, and all the time in my Rust code. Python punishes you for threading. The GIL makes things feel atomic when they aren't, reduces you to concurrency, all while paying the cost of an OS thread. Threading primitives are also quite weak, imo, and multiprocessing is a dead end.
And, really, Python is anti-optimization, which is a good and bad thing, but it's a bit extreme.
* Implicit exceptions everywhere. I see 'raise Exception' defensively placed in every Python codebase because you never know when a piece of code will fail. I see a lot of reliance on 'retry decorators' that just catch everything because you never know if an error is transient or persistent.
The common "don't use exceptions for control flow" is broken right off the bat with StopIteration. I just think error handling in Python is god awful, really.
* Mypy is awesome, but feels like a bit of a hack. The type system is great, and in theory it would solve many problems, but coverage is quite awful in open source projects and the type errors I get are useless. I actually only use mypy to make my IDE behave, I don't run it myself, because my types are actually technically unsound and the errors are too painful to work with (and honestly the type system is quite complex and hard to work with, not to mention the bugs).
There are lots of other smaller issues, such as my distaste for pip, py2/3 breakage, lack of RAII, the insane reliance on C all over the place, I think syntax is bad in a lot of places (lambdas, whitespace in general), etc but these are probably my main sticking points with using the language for large projects.
Again, Python is actually my favorite language for specific tasks, but I really think it's best optimized for small codebases.
The common "don't use exceptions for control flow" is broken right off the bat with StopIteration. I just think error handling in Python is god awful, really.
That's not an idiom python has ever subscribed to though, it's always subscribed to the "Better to ask forgiveness than permission" - I think there are strengths to both viewpoints, but honestly I think "don't use exceptions for control flow" is more of a convention that a "truth"
> honestly I think "don't use exceptions for control flow" is more of a convention that a "truth"
In absolute terms or when coding on paper, perhaps. But in the real world and if performance even remotely matters, it’s as close to a universal rule all languages end up embracing or turning into creaking hulks of slow code given how exceptions work in practice at the level of cpu execution units.
C#/IL/.NET embraced excerpts heavily at around the same time (start of the 2000s) but in time developers (in and out of Microsoft) learned the hard way that it doesn’t scale. With .NET core, exceptions for flow control are completely verboten and APIs have been introduced to provide alternatives where missing. Exceptions should be so rare if you chart all handled exceptions, you shouldn’t see any and would thoroughly explore why one or more pop up when a system or dependency hasn’t exploded.
If you care that much about performance you probably shouldn't use Python in the first place. Python deliberately prioritize ease of development above performance.
I'd be interested to learn about how exceptions are typically using in Ocaml for non-local control flow.
Exceptions have two core properties: (1) non-local jump (carrying a value) and (2) dynamic binding of place to jump to. Contrast this with e.g. break / continue in loops where (2) does not hold. If most use cases of performant OCaml exceptions were not making use of (2) that would be an interesting insight for programming language design.
No data as such. But here's the first example in the "batteries included" list library:
let modify_opt a f l =
let rec aux p = function
| [] ->
(match f None with
| None -> raise Exit
| Some v -> rev ((a,v)::p))
| (a',b)::t when a' = a ->
(match f (Some b) with
| None -> rev_append p t
| Some b' -> rev_append ((a,b')::p) t)
| p'::t ->
aux (p'::p) t
in
try aux [] l with Exit -> l
Here, the Exit exception is being raised as an optimisation: if no modification happens, then we return the original list, saving an unnecessary reverse. The try is the only place that the Exit exception is caught, so the jump location is static, much like a break.
That's, as you say, a statically scoped fast exit. One does not need the full power of exceptions for this (exceptions'd dynamic binding comes with a cost). If exceptions are widely used for this purpose, one might consider adding a nicely structured goto to the language. Something like linear delimited continuations?
Sorry, I misunderstood what you were after. Ocaml exceptions are used more generally, and often make use of 2.
For instance, the basic stream/iterator in Ocaml is Enum, which always uses an exception to signal when the stream has been exhausted, rather than providing a "has_next" predicate.
That is interesting. Once could argue that since fast exceptions contain non-local jumps with a static target, it is enough to supply only the former, so as to simplify the language.
Interestingly .Net has very performant exception handling as well. Not sure if something changed internally with Core, but using exceptions in place of return error codes was incredibly common within the .Net ecosystem.
.NET has always had slow exception handling because it tied/ties in Windows’ Structured Exception Handling (SEH); that’s rather slow but provides e.g. detailed stack traces even for mixed-mode callstacks.
Having ported some decently large codebases from OCaml to F#, the heavy use of exceptions in OCaml (where exceptions are very lightweight by design) had to be changed to normal control flow with monads to achieve good performance in F# — specifically because of .NET’s slow exception handling.
> The common "don't use exceptions for control flow" is broken right off the bat with StopIteration. I just think error handling in Python is god awful, really.
Python explicitly does not agree with this...
The same for a lot of your other complaints. It sounds like you are trying to write some other language in Python. Similarly if someone in a team using Java tried writing Python in Java they would complain a lot and end up with ugly hard to work with Java.
Python's exception system isn't bad. It's way ahead of Java's or C++ exceptions. The gyrations people go through in Go or Rust to handle errors are far worse. The exception hierarchy lets you capture an exception further up the tree to subsume all the more detailed exceptions. (Although the new exception hierarchy in 3.x is worse than the old one. In 2.x, "EnvironmentError" was the parent of all exceptions which represented problems outside the program, including the HTTP errors.[1] That seems to have been lost in 3.x).
I think StopIteration is being removed. That was a a bad idea.
> If a StopIteration is about to bubble out of a generator frame, it is replaced with RuntimeError, which causes the next() call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception.
OK, I'm willing to contend that that is the case. It doesn't have much to do with my overall issue with error handling.
> The same for a lot of your other complaints. It sounds like you are trying to write some other language in Python. Similarly if someone in a team using Java tried writing Python in Java they would complain a lot and end up with ugly hard to work with Java.
This is a very loose criticism of my post, I don't know how to respond to this. I've written Python for years, I think I gave it due credit for what it's good at.
I am not trying to write some other language with Python, I just think Python is not a very good language compared to others given a lot of fairly typical constraints.
Why? Of course it's entirely possible to build programs of a larger size. I'm just saying that after a certain point you start hitting walls. There are tons of ways to hedge against this - leveraging mypy early on, spending more money (on eng, infra, ops, etc), architecting around the issues, diligence, etc.
It would be very silly for me to say that you can't build large systems in Python, I've worked on plenty that are much larger myself.
Saying a language is possibly the best for two important use cases (exploration, small programs) is quite a statement, in my opinion. I don't think I believe that there's a language that excels so well in, say, building web services.
Whats the convincing argument for defending this? Ive always heard its just the way python is. Through experience using exceptions for anything other then exceptional situations and errors seems messy.
I suppose that comes back to use cases. I am one of the 'getting things done crowd', rather than a computer scientist. A lot of my work had been in things like EDI, or similar integration code.
Imagine you are working through a series of EDI files and trying to post them to a badly documented Rest(ish) API of some enterprise system. If the file is bad (for whatever reason) you need to log the exception and put the file in a bad directory.
Pythons use of exceptions for control flow is perfect for this. If file doesn't load for whatever undocumented reason, revert back to log and handle the fallout.
"Oh I see a pattern, this API doesn't like address lines over 40 characters, I will add a custom exception to make the logging clearer, and go and try and see if I can fix this upstream. If not I will have to write some validation"
It is this dirty world of taking some other systems dirty data and posting it to some other systems dirty API that I find Python rules.
I have never worked on a large application where I owned the data end-to-end. Maybe there are better choices than Python for that?
My feelings exactly. I write a lot of kinda scientific code that is not really computational but takes one or more horribly messy datasets and combines them to do some analysis. I now routinely use exceptions to filter out bad/complex data-points to deal with later.
Data scientist here. It’s great because non programmers find it easy and we have so so many tools available.
That said, the amount of cpu cycles wasted even with best of breed pandas is insane, and when you want to do something “pythonic” on big data it all falls down. When you want to deploy that model you are also going to have problems.
That said, it’s still the best tool for the job, but it’s certainly not because of the creators of Python.
My opinion on static type systems is that, unless you're actually doing a more type driven development style and leveraging the type system, the single greatest benefit is IDEs will autocomplete for you, give you 'jump to' etc.
Python without type annotations can be painful in an IDE - it chokes a lot on those features.
Since I don't take a very type driven approach to Python (it would be too slow since I'd have to figure out 3rd party library types and shit like that) I just write annotations in places where I personally know the type. Mypy complains about this for various reasons - probably because my annotations are not technically correct, because I'm not a pro at generics in mypy and working out inheritance, as well as general mypy issues like poor support for class methods and that sort of thing.
But I ignore all of those errors because the IDE can still work things out.
What you’re describing is what happens when you have a developer who’s bad at their job.
I see the point that python invites these anti patterns.
But on the other hand, a software developer that returns Union(list[item], item]) in Python is probably also going to mess up a Java program sooner or later.
I don't think you'd ever write that in Java because Java would punish you for it. You can always blame the programmer, but I'm always going to blame the language for making 'bad' easy.
The overhead obviously is still there, but the interface is a drop in replacement for threadpoolexecutor, which looks basically just like a multithreaded/async/future-based collect.
The interface may have improved, I haven't paid attention to mp in a long while. If I'm reaching for mp I generally just accept that I'm using the wrong language.
I have a hard time reasoning about multiprocessing code, for various reasons. It's bitten me in a lot of weird, distinct ways, not really worth listing here.
To communicate across mp you use pickle, which is a pretty significant performance impact relative to something like threading. There's also the issue of copy on write memory + reference counting interacting poorly.
I suspect this is the root cause of the difference in our experiences. My uses of mp have usually been somewhat more "embarrassingly parallel", for instance having a list of data elements which need to be processed with the same algorithm. For this use case, the usage of mp is pretty simple, often only a `pool.map(f, xs)`.
I can imagine that pickle might have tricky edge cases and/or be slow.
1) No pattern matching. In 2019, this is just unacceptable. Pattern matching is so fundamental to good FP style that without it, it becomes almost impossible to write good and clean code.
2) Statement vs expression distinction. There’s no need for this and it just crippled the language. Why can’t I use a conditional inside an assignment? Why can’t I nest conditions inside other functions? It makes no sense and is stupid and inelegant.
3) Related to 2), why do I need to use a return statement? Just return the last expression like many other (better) languages
4) Bad and limited data structures. In Python all you get are arrays, hashmaps, sets. And only sets have an immutable version. This is unacceptable. Python claims to be “batteries included” but if you look at the Racket standard library it has like 20x more stuff and it’s all 100x better designed. In Scala you get in the standard library support for Lists, Stacks, Queues, TreeMaps, TreeSets, LinearMaps, LinearSet, Vectors, ListBuffers, etc.
5) Embarrassing performance. Python is so slow it’s shameful. I wrote a compiler and some interpreters in college and I honestly think I could create a similar language 10x faster than Python. Sometimes you need to trade off performance and power, but that’s not even the case with Python: it’s an order of magnitude slower than other interpreted languages (like Racket).
6) Missing or inadequate FP constructs. Why are lambdas different from a normal function? Why are they so crippled? Why do they have a different conditional syntax? The only sort of FP stuff Python has is reduce/filter/map. What about flatMap, scanr, scanl, foldl, foldr? Or why doesn’t Python have flatten? All of these are very useful and common operations and Python just makes everyone write the same code over and over again.
7) No monads. Monads can be used for exceptions, futures, lists, and more. Having to manually write some try catch thing is unseemly and worse than the monadic Haskell or Scala approach.
8) No threads and no real single process concurrency. Despite Python being used a lot, no one really seems to care about it. How can such a problem not be solved after over 20 years? It’s shameful and makes me wonder about the skill of Guido. There’s no reason why Python couldn’t have turned into something beautiful like Racket, but instead it has been held back by this grumpy old guy who is stuck in the past.
9) Others might not have a problem with this, but I detest Python’s anything can happen dynamic typing. It makes reasoning about code difficult and it makes editor introspections almost impossible. If I can’t know the exact type of every variable and the methods attached to it, it hampers my thinking a lot. I use Python for data science and if I could just have a language that was compiled and had static typing I would be 3x as productive.
Let me conclude by saying there currently is one good reason to use Python: if the domain is ML/DL/quant/data science Python is still the undisputed king. The libraries for Python are world class: scipy, sklearn, pandas, cvxpy, pytorch, Kerala, etc.
But Julia is catching up very fast and the people I have talked to are getting ready to ditch Python in 2-3 years for Julia. I don’t think I’ve encountered anyone who didn’t prefer Julia to Python.
It seems you just want a functional language. If so, why are you using Python, as there are much better alternatives like you mentioned, if not in the data science space? I use Python only for data science and generally Elixir, Racket, Haskell etc for other use cases.
> and that infernal pep-8 linter insisting 80 characters is a sensible standard in 2019
I don't know why 80 characters is a problem. I don't use the linter but I enforce this rule religiously with a couple of exceptions (long strings comes to mind). It forces me to think heavily about the structure of the code I'm writing. If I'm nesting so deeply, something has gone wrong. If I've got a ton of chained methods or really lengthy variables, it forces me to rethink them.
This also has the advantage of being able to put 4 files next to each other with full visibility. Vertical space is infinite, horizontal space isn't. It's probably a good idea to use it.
It's also awesome if you have to do code reviews on a laptop or don't have a massive screen available.
That said, we usually just go with autoformatting via black, which is 120 by default. No more hassle manually formatting code to be pep8-compliant. Just have black run as a commit hook, which is super easy via pre-commit [0]. And you can run the pre-commit checks during CI to catch situations where somebody forgot to install the hooks or discovered `--no-verify`.
Can't really imagine developing Python without Black any more.
I haven't got round to trying Black, but according to the project's README[0], the default is 88. Personally I think 79 is fine, but I can cope with up to about 100. Above that and you risk some really crappy code in my opinion.
EDIT: Sounds like the Black author agrees. "You can also increase it, but remember that people with sight disabilities find it harder to work with line lengths exceeding 100 characters. It also adversely affects side-by-side diff review on typical screen resolutions. Long lines also make it harder to present code neatly in documentation or talk slides."
While initially resistant I've come around on Black for our team and a failed Black check will now make a CI build fail for all our projects.
We're still using the community edition of SonarQube [0] for inspection but Black finally did away with the constant bikeshedding over formatting minutia, seems like it's saving us tons of time.
All I know is that with 80-char width I can have 2 files side-by-side on a 15" MBP along with the dir-tree on the left in an editor like PyCharm or VSCode and fully see both files wo wrapping. It helps my productivity immensely.
Same deal when it comes to reviewing PRs in GitHub. Wrapping just interrupts flow for me.
I feel the complete opposite. I really enjoy working with python over any other language. R does linear models and time series better and matlab has its charm, but overall I prefer python. Python is so easy to read and quick to program in. I am so glad I am not in the Java/C++ world anymore, but I know people in different roles have to deal with different issues.
> I really enjoy working with python over any other language.
I assume you mean, "over any other language I have tried" ?
As someone with a mathematical background myself, I am always surprised at how many data scientists and quants are ignoring more mathematically principled languages like F#, OCaml and Haskell.
> What does it mean? Have you done it in any of those language?
I did. I'm doing a image processing recently and use OCaml for prototyping. I've tried python (I've used it a lot for that long time ago), I've failed, it felt to awkward. I've described my experience here [1]
If you have no experience whatsoever with ML family [2], and doing all the stuff in python, you'll most likely be much more productive with python of course.
But I find ML-like languages way more pleasant, and I'm far more productive with libraries like owl [3], which are more fundamental and don't have fancy stuff, and ML, rather than with python and fancy lib like numpy/scipy.
Also Julia could be a good choice hitting a sweet spot between fancy libraries and fancy language.
Right now I’m experimenting with a pretty complicated model (60+ layers of multiple types), and I plan to train it on several hundred GB of data, using 8-16 node cluster (4 GPUs per node). Does Owl have a well tested and well documented autograd library with distributed GPU support (e.g. Horovod)? With a good choice of optimizers, regularizers, normalizers, etc, so I can focus on my model and not on debugging the plumbing or implementing basic ops from scratch. And last, but not least, it must be as fast as TF/Pytorch.
If the answer is “no”, then it does not matter whether I’m an OCaml expert, because I’m still going be more productive with Python.
p.s. Julia is nice though, hopefully it will keep growing.
I feel what you're saying is that regardless of how subpar a language is compared to alternatives as long as it has community built specific libraries that solve your problems you're more productive using them than anything else.
Which is of course a fair point. A language by itself is probably not even in the top 3 considerations when choosing new tech. Stuff like runtime, ecosystem and the amount of available developers would probably be more important in most cases.
> A language by itself is probably not even in the top 3 considerations when choosing new tech. Stuff like runtime, ecosystem and the amount of available developers would probably be more important in most cases.
Totally depends on a domain. In serious mission critical software you wont use libraries, but will use the language.
Yeah I don't disagree. But even there you would have similar other considerations besides the language. Like most still end up with C/C++ there even though there are others like Crystal, Nim, but you just don't find developers who know them easily, nor do you have any ecosystem support.
> Like most still end up with C/C++ there even though there are others like Crystal, Nim,
Because C++ and C are significantly better than Nim and Crystal.
There are also Ada and Spark and aerospace and very critical stuff.
> just don't find developers who know them easily
We don't look for OCaml/Ada developers, we hire programmers, and they program OCaml and Ada. It's not a big deal for a good programmer to learn a language, especially while programming side by side with seasoned programmers.
In my 6 years with Python, the only dissatisfaction with the language I felt was from parallel programming. I switched to Python from C, and at the time, I missed C transparency and control over the machine, but that was compensated by the Python conciseness and convenience. Then I had to dig into C++ and I didn't like it at all. Then I played with CUDA and OpenMP, and Cilk+, but I wished all that would be natively available in a single, universal language. Then I started using Theano, then Tensorflow, and now I'm using Pytorch, and am more or less happy with it. It suits my needs well. If something else emerges with a clear advantage, I'll switch to it, but I'm not seeing it yet.
As a bonus, it IS Python (numpy) in the background mixed with Scala. So you can use each language where they make the most sense - Python for the maths number crunching and Scala for the business logic and the architecture.
I think Spark also has .net bindings (so you can also tick F# on that list...).
As much as I love the languages you mentioned: I think it's a major weakness of them that they don't have the linear algebra libraries integrated such that you can do this the same way Python does.
For those unaware: Haskell has a REPL (ghci), and you can make files more script-like with the (currently most popular) build tool stack[0] if you include:
It's language vs libraries. If you have a library that has a function
get_the_shit_done_quick ()
than you don't care much about the language.
When you don't have such function, you need an expressive language to write it (and a bulk of python libs are not written in python, tho mostly for the performance reasons).
So it's all about finding a sweet spot between fancy libraries which do the shit for you, and fancy language, which let you to express things, absent in libraries.
This sweet spot differs from domain to domain, from user to user. Even in numerical stuff someone could have a requirement for a better language, although this domain is indeed to well defined to have enough fancy libraries.
Language vs libraries isn't just about an expressive language to build in when you don't have a library. The likelihood of a library's availability also depends on the barrier to entry. An amazing language that isn't usable by biologists won't have many libraries that solve biologist's problems.
To your original point of being "surprised at how many data scientists and quants are ignoring more mathematically principled languages like F#, OCaml and Haskell," I'd much rather use one of those languages, but I'd have to build the foundations myself. Today, they aren't the right tool for the job. They don't have the libraries I need, which means I don't build further libraries for them, making other people less likely to build on them, so they aren't the right tool for the job tomorrow either. I'd say it's a network effects thing primarily.
Well, yeah compared to R and matlab, I am willing to believe Python excels, but the person you are replying to is probably not doing data science, so he has options besides the 3 just mentioned.
Re. linting, I'd highly recommend Black with whatever line length you want - it'll reliably reformat your code, and once you lose the urge to reformat while typing it's fantastic. It's like deleting a bunch of useless mental code. And the code reviews are even better: include a linting step (ideally with isort) in CI and you can avoid 95% of formatting comments.
100% this. I just switched to using Black recently and not having to ever fix a lint issue again has been life-changing. Use Black with pre-commit (https://pre-commit.com) and never look back.
Over-sensitive individuals are hard to please and often unhappy. That's more of a personality flaw than a flaw with the current state of software engineering.
If there's one thing wrong with our profession is a lack of ethics and accreditation - we're essentially letting random people build critical infrastructure and utilities.
We don't have a tooling problem, in fact we have too many tools.
I see so many people (especially on HN) fixating on tools, dissecting programming languages, editors and libraries into their most minute parts and then inevitably finding faults in them, disavowing them and swearing that other tools X, Y and Z are purer and worthier.
If you want to stop hating software and improve your burn out, stop caring about irrelevant stuff.
Is that supposed to help someone? I fail to see how telling someone "just ignore the stuff that irks you that you have to spend 40+ hours a week dealing with. You are overly sensitive and your concerns are irrelevant" helps anyone. Even if it was true, it was delivered in such a ham-fisted manner that I can't imagine anyone taking it to heart.
I sometimes have a tendency to focus only on the negative aspects of some things, while ignoring that all in all, those things are fine. I don't think I'm alone in that, certainly not in our line of work.
A call to "snap out of it" seems that it can help in such situations. Python is not a programming language that should make people burn out or angry. Very few languages should be capable of that, so I think this issue goes deeper than just flawed tools.
I find that the only way not to go nuts in this profession is to ignore most of it and most of it is really not relevant to building good software. There are just too many tools and always searching for the perfect tool is a recipe for being unhappy.
>If there's one thing wrong with our profession is a lack of ethics and accreditation - we're essentially letting random people build critical infrastructure and utilities.
As someone with a P.E. license that spent hundreds of hours studying and had to sit for that excruciating 8-hour exam twice, I don't think even 5% of the software developers in the US could pass an equivalent exam. Granted, I think the sector I took it in has a harder test than some, but it is a weed out test for someone that already took 4 years of Engineering school.
Some takeaways from this:
1.) I learned a little bit from studying, but overall even though it was hard, I would've learned a lot more by getting a Master's degree.
2.) The test isn't good at determining if you're good at your job or honestly even minimally competent in your area. For example, even in a specialized field (power systems engineering), there are thousands of diverse jobs (power generation, distribution, electrical contracting, power transmission operations, power transmission planning....etc etc) so the test only had a few questions on my particular area.
3.) There are a lot of amazingly smart people working in software, but the skill range seems to be bigger than in traditional engineering fields where most engineers are fairly similar in skillset (there are some outliers) as they all generally have to pass the same courses if their university is accredited (talking about USA here). In the software world, you have software engineers and computer science doctorates mixed with someone with very little training that is trying to wing it. That means the dataset has a far greater range on skillsets. One employee might have had a class on compilers while another just learned what a for-loop is. In engineering, we generally all show up to the first day of work with the same building blocks (thermo, statics, Dynamics, circuits, differential equations, stats, calculus, basic programming...etc). The only difference between me as a senior engineer and a new hire is 9 years of experience, knowledge of the business and tools and ability to get things done without oversight. It makes a big difference, but I wouldn't expect them to be lacking any of the tools or training that I have picked up (ok...maybe databases).
I'm struggling a bit to convey the overall message that software engineering seems a bit different and licensing would therefore need to be different if done. Perhaps you could have licensing for individual subjects? For example, you could pass a basic test on relational databases where you prove you can do basic operations such as select, where clauses, joins, updates, exports...etc. Then you'd have another to prove you were minimally competent in Java? Would that be of any value to an employer? I don't know. I'm guessing someone already does this for Oracle and Java too.
so I am studying for the FE (I need a lot of math before taking it is realistic) mostly 'cause it gives me this broad feel for things Engineers all know. (I will take the 'other disciplines' - mostly because I want this to be as broad as possible; being broad but shallow makes it a lot easier for me, too, but for me, it being broad is an advantage in other ways, too.)
I personally find tests to be way easier than school, and the schools with reputations that are worth something are... pretty difficult for people like me (who weren't on the college track in high school) to get into. (and there is something of an inverse correlation between the prestige of a school and how flexible they are about scheduling around work; especially for undergrad)
From what I've seen of the test, it does provide some foundational ideas of what engineering is about. Like, it goes a lot into figuring out when things will break - something I haven't really seen a lot of in software engineering.
What I'm saying here is that I dunno that an optimal SWE PE would test you very much on the specifics of Java or SQL or what have you. I mean, from my experience with the FE, at least, they give you a book that has most of the formulae you need... and you are allowed to use a copy of that book during the test, you just need to be able to look it up and apply it. Seems like they would do the same with Java or SQL.
(I mean, to be clear, to apply the formulae, you still need to have more math than I do. I imagine the same would be true of sql or java, only I'm pretty good with SQL, having 20 years of propping up garbage written by engineers who weren't.)
From what I've seen of the software engineers, Most of the time, the software guys throw something together, toss it over the fence and move on. Clearly, they didn't do any analysis to figure out how much load the weak point in the system can handle, or even what the weak-point was. It's incumbent upon me (the SysAdmin; usually the least educated person in the room) to feed load to it at a reasonable speed and to back off and figure out what happened when the thing falls over.
I mean, I think the real question people are bringing up here is "what if we treated software engineering, or at least some subset of software engineering more like real engineering?" - like clearly sometimes you can slap together something over the weekend and it's fine, but... if you are building a car or an airplane or a tall building or something, you probably want to put in the time to make sure it's done right, and for that, you need rules that everyone knows; the PE system, I think, establishes rules everyone knows, while I think software engineering mostly works on the concept of "it worked for me"
Wait...are you studying for the FE without getting an engineering degree? Props to you. One thing to keep in mind though is that there is a morning and afternoon session that are both 4 hours iirc. The first session is always a general session which covers the absolute basics of Engineering math, circuits, thermodynamics, statics, Dynamics, chemistry, and physics. It really is very easy if you remember the classes. Some of the circuits problems can be done in your head and the statics problems might have the resultant force being part of a 3-4-5 right triangle (again, it shouldn't take much thought). The purpose of this is to ensure you learned the absolute bare minimum in these classes. One reason the general questions have to be easy is that depending on your course schedule, it might have been two years since you took a course (Ex: you might have taken only one thermo class as an electrical engineer during your sophomore year). The afternoon test is either also general or specialized to a discipline (Ex: chemical engineer) and are much more difficult in comparison. I barely studied for the FE and felt I knocked it out of the park (especially the morning session). I spent months of all my free time studying for the PE and failed the first time...it is difficult. Keep in mind that both of the tests have a lot of breadth, but little depth. Going into an actual Engineering curriculum will teach you a whole lot more. MIT used to (might still) have a circuits Edx class online for free which covers the first out of 3 circuit classes a EE will take...that should help a little with the scale.
Software is weird as the hardware, languages, and frameworks are always changing and the optimal work done on any project seems to be just enough to keep the customers from going to a new product and not necessarily the best or even a good product in many cases. There are cost constraints in Engineering as well (good, fast, & cheap...pick 2), but it still feels pretty different to me than software engineering where something breaks all the time in any non mainstream software I've ever used.
Yeah, they'll let anyone with a couple hundred bucks sit for the FE. The chances of getting a PE or even an EIT, though, without a degree are... slim to none. But that's not really my goal? (I mean, it would be if it were just tests) Mostly I just want to know those 'absolute basics' of which you speak, and I like to have a test to study towards.
I'll check out that edx class, thanks, that sounds like my thing.
I dunno, man. I own a bunch of stock in BA right now. I bet that the idea of spending twice as much on software engineers and hiring licensed folk to write their software is looking pretty good to Bowing execs right about now, even from a plain profit and loss perspective.
(of course, a lot of professional engineers were involved in building that plane... but it's pretty unlikely that there were any PE software engineers involved, just 'cause there aren't many. Would that have helped? maybe, maybe not. to the detail that I've studied (not.. really all that much) it sounds like they made some really basic mistakes, like relying on a sensor that wasn't redundant, and those mistakes should have been caught by the other engineers. I don't know that it was entirely a software engineering problem. )
As software mistakes get more and more costly, it starts making more and more sense for execs to hire people who are properly licensed to supervise the project. (I mean, assuming such people exist, and for software engineering, right now, you could say such people don't exist.)
Oversensitive? Is the contractor who chooses a Dewalt or Ridgid over a Ryobi for daily work "caring about irrelevant stuff"? A drill is a drill right? Why is it different for us in software?
Maybe. (and I thought festool was the new hotness? I thought that at least rigid and ryobi had been devaluing their brand by selling cheap junk under their brand? But I'm solidly in the 'prosumer' and not 'real contractor' end of the market, so not an expert or even close observer, really.)
But I think the point OP was making is that contractors have a licensing and training program, and if you hire someone to put a roof on your house, they either have to go through that process or work under someone who went through that process. I mean, choosing the right tool is a small part of that, but someone in the chain of command is gonna have words with you if you bring your Xcelite screwdriver and try to use it on the roofing job.
That's not true almost anywhere in software, and that probably makes a big difference.
(I mean, not being educated myself, i personally have mixed feelings about licensure. But it's a good discussion to have, and I think that there should be something like the PE for software engineers (and there was, in texas, but it will be discontinued this year. )
I don't know anything about those brands you mentioned, but a programming language is more complex than most (all?) mechanical tools, it's designed over years and continues to change over decades.
It's impossible to do a perfect job under such conditions, and it's anyway impossible to please everyone.
> Over-sensitive individuals are hard to please and often unhappy. That's more of a personality flaw than a flaw with the current state of software engineering.
Ah yes. Just remove the part of myself that got me into software as a child, and proceed robotically.
> If there's one thing wrong with our profession is a lack of ethics and accreditation - we're essentially letting random people build critical infrastructure and utilities.
Spot on. Leftpad (and perhaps the whole js ecosystem) are good examples.
Those differences don't matter much when it comes to building software that fulfills specific requirements.
I am a big fan of compile-time checking, but there's a lot of good software built without it and sadly there's also a lot of successful slow software. These are disadvantages, not an impassable barrier.
I know I’m responding to opinion but: developer productivity isn’t one of the pitfalls of Python.
Yes I agree that if you’re using Python for a large scale project involving lots of developers its not the best; but that’s because it doesn’t have a good type system.
You can’t work out why people like it so much because of this misconception. The languages that you gave as examples most definitely do _not_ value your productivity, it values correctness as enforced by a type system and refactoring needed for large projects.
I am more productive in a language with an expressive type system (e.g. Haskell) than one without. Thinking about types not only guides me towards a working solution quickly, but the checker catches all my minor mistakes.
In Haskell, you can actually defer all type errors to runtime if you want to. But I have never felt this makes me more productive.
There are plenty of real world situations where neither the requirements nor the prerequisites value correctness, but getting an 80% or even 99% correct set of results while flagging the unhandled cases is very valuable.
I'd say the most immediate value/payoff of correctness is ensuring your own code is consistent with what you think it does rather than correct wrt to some sort of external specification.
The bigger the team, the more distributed the engineers and the bigger the codebase, the more productivity is lost by using scripting languages. I find it infuriating because it is so frustrating and it definitely does not feel productive.
For a lone hacker, its the other way around. Compare e.g. to Golang's "programming at scale" rationale.
The way I think about it is that Python is a strong local optimum in a space that has a massively better optimal solution really close by. But it's nearly impossible to get most people's algorithms to find the real optimum because Python's suboptimal solution is "good enough". And the whole software industry (and in some ways, by extension, all of humanity ... to be over melodramatic) is suffering for it.
> And the whole software industry (and in some ways, by extension, all of humanity ... to be over melodramatic) is suffering for it.
I don't think the biggest services built with Python (think Instagram, Dropbox, etc.) have more consumer-facing issues than services written in other languages.
If you're talking only about developers, fine, however I also think most Python developers like the language. For me it seems that Python has strong vocal critics, that show well in places like HN, however it is not representative of the majority.
So I really don't think Python is making the humanity suffer, for any good measure of suffering.
There is great saying that Python is the second-best language for everything and it might be true. Where Python excels is that you have good-enough libraries and support almost everything. All other languages have some pain points where they shine very brightly in some areas but they have very bad or non-existing libraries and support to other areas.
```--max-line-length=120``` passed to flake8 will switch it to 120. Use another number for a different length.
Things like max line length should be something your org or your fellow contributors decide on, not something dictated to you no matter what the language.
A corporate point of view is than any programmer should be interchangeable with the minimum amount of fuss. I understand why someone building an injury organization has a responsibility to think about the future.
But I confess that sometimes I feel very demoralized when an organization implicitly tells me that my years is study and my ongoing self-education and practice is all meanless Because in principle someone fresh out of college should be able to take over my project with the two week handoff and a few docs.
> Every moment working with python (and that infernal pep-8 linter insisting 80 characters is a sensible standard in 2019) increases my burnout by 100x.
You hyper-fixated on that tiny thing. It increases your burnout 100x, remember?
It's the only problem you mentioned, and I inferred from your post that it was really bothering you. That was my thinking anyway.
I see now that you mentioned other issues in other comments.
Anyway I appreciate you sharing your feelings to an evidently unreceptive audience. It's nice to know there are better ecosystems out there waiting when I get a chance to look for my own next step.
i fully agree with you. it has a stubbornness about it and not in a good way like the stubbornness you might find in lisp, scheme, and sml derivatives. i have had to write python on only a few side projects, but it was miserable, as you say. not only is the language all over the place and embarrassingly slow, the ecosystem is as well. i tried getting an application someone left when they left running on windows 10, and it was basically a null effort without a complete rewrite, upgrade to 3.x, and upgrading/replacing GUI libraries.
if i had to write python as a day job, i would quit. i have said it before, but python is the new c++, helping people everywhere write brittle, unmaintainable code.
I'm sympathetic that Python is relatively not a great language, but IMHO an 80-character line limit is quite reasonable. It's easier to read on smaller screens, easier to view side-by-side diffs, and tends to force you to break up your code more.
That said, this shouldn't be a lint, it should just be enforced by a formatting tool as a format-on-save setting. It just destroys all the wasted arguments about formatting and the wasted time trying to manually line up code.
I'd also add that while perhaps a bit on the pessimistic side, I tend to view the 80 char rule / limit not as an ancient hardware limitation of monitors, but as a limitation of our eyes and visual processing circuitry. There is a reason why newspapers and well laid-out websites don't have 300 char width lines. Those are physically harder to read, whether we want to admit it or not, as our eyes lose track of the flow to the next line.
I'm all for decreasing unnecessary cognitive load, there should be quite enough of that without us adding more by accident.
I don't understand–I quite enjoy reading a well-formatted plaintext email that sticks inside an 80-character line width. Any mail client worth its salt should be able to display that.
> as a limitation of our eyes and visual processing circuitry.
If so, then why not put the limit on line length without trailing whitespace? Because it makes no sense that with indentation I should lose available characters.
> There is a reason why newspapers and well laid-out websites don't have 300 char width lines.
Yes, and the reason is, print and transportation are expensive, so newspapers found a way to cram as much text as possible in as few pages as possible. You don't see them replicating this style on-line, and neither you see it in magazines that are priced well above their production & distribution costs.
The reason "well laid-out websites" don't have 300 char width lines is because web development is one large cargo culting fest of design. 300 may be a bit much due to actual physical length, but your comment has 200+ on my machine and reads just fine.
I don't buy these "80 chars / short lines and ridiculous font sizes are optimal for the eyes" arguments. They fly in the face of my daily experience.
It's hypothetical so I can't say for sure, but I did work a lot with the JVM and I used Clojure (and I'd probably use Kotlin now as well) and I didn't feel as burnt out as I do now.
I have a acquaintance that swears by Clojure (for data modelling of sorts) and specifically that I should also use it (due to my category theoretic background).
Yeah, I'm a bit sick of Python. We have some internal utilities written in it and while developers could manage all the dependencies and everything, we kept having problems with getting everything working on the machines of our electronics techs.
Gradually pretty much everything has been rewritten in C++ (with Qt GUI framework). Way easier to be able to have a setup program (using free InnoSetup) that just extracts the eight or so DLLs required in the same folder as the EXE and that's it.
We just use Python for a bit of prototyping and data crunching here and there now.
> Way easier to be able to have a setup program (using free InnoSetup) that just extracts the eight or so DLLs required in the same folder as the EXE and that's it.
I have 3-4 files side by side, Everything else is max 80 chars. I really don't understand why people need more, because long lines are REALLY HARD to read.
You mean like 64 chars, after you account for typical indentation? I hit that limit way too often when using descriptive names.
I usually view two files side by side per screen (so 4 in total). I sometimes up this to 3 per screen, but the trick here is to use a smaller font. Right now, if I split my editor into 3 panes, each has 98 columns available.
I guess it can happen, but in my experience, I had more problems with overly long names hurting readability: something like ExampleDataUpdaterFactoryConfigReader. (No, I don't think IDE makes them acceptable because you still have to read them in the code, IDE or not.)
Of course, 80-character limit doesn't guarantee good naming, but it acts as a friction against adopting ultra-long names, occasionally forcing devs to find a better alternative. YMMV.
I work at the same place you do. I code often enough from my macbook, using likely the same tools that you use, and they're truly painful with >80 character lines. I don't understand how Go and Java devs survive.
I also don't think an 80 character line limit is a bad thing: small well defined functions that do only one thing are good. Long lines often encourage people to want to nest code deeply (which is terrible!) or to write complex one-liners, and that + list comprehensions is a dangerous pairing.
Java dev here. I would try and defend a 120 character limit but I have a 34" 5k widescreen monitor so I probably don't have a leg to stand on.
I'm writing a mix of Java and python at the moment. Python for lambdas behind an API gateway and Java on some containers for stuff that's more complex but evolves more slowly.
It's neither python or Java where I'm really spending my time though. It's CloudFormation, Ansible, Jenkins and stitching the whole system together for CD that's killing me. I feel like programming languages are the easy bit these days.
> I feel like programming languages are the easy bit these days.
Agreed. The mainstream garbage-collected languages are all basically the same in the grand scheme of things. The work that takes most of my time (and growing) lately is packaging, testing, deployment, etc.
The only tool that gets remotely irritable about it is the one we use for diff reviewing, afaict?
> I also don't think an 80 character line limit is a bad thing: small well defined functions that do only one thing are good. Long lines often encourage people to want to nest code deeply (which is terrible!) or to write complex one-liners, and that + list comprehensions is a dangerous pairing.
Python already starts you out at a nest of 2-4 characters, so we don't even get the full 80.
But honestly I don't think a 100 character line is going to doom us all to hyper-nested expressions.
Well that sounds like a personal problem and it has nothing to do with the language. As a counter-anecdote, I've been working on Python projects for over a decade and have yet to experience the draconian linter settings you described.
Thanks for posting. I've wondered if it's just me or does anyone else want to leave software engineering because of Python's dominance. There's no arguing against it either because as this thread shows, "I like it for my use case, look at these libraries that let you get X and Y done SOOOO easy, so it must be great for everything."
I hated Javascript back when all it had were callbacks, but once native promises stabilized, I loved the simplicity + ubiquity of the promise abstraction, and then later async/await. Now it's actually my favorite dynamically-typed language.
I've noticed a lot of Javascript hate also comes from people just disliking client development (which isn't easy on any platform).
I thoroughly enjoy UI/client development, but the web app tooling (layer upon layer that is supposed to "fix" HTML/CSS/JS every few years) is frustrating to me. Thus, I avoid it like the plague. :)
To be fair, the JS ecosystem I think is a bit more tolerable than Python. JS isn't being promoted far and wide as the good-enough tool for everything like Python is. A lot of JS is focused on the "view", and I can see a dynamic language fitting there (though React/Redux is making me rethink that).
It is a language, better than MatLab/R/SPSS, which come before it.
I honestly don't think it is that bad. And they are many people don't care from a programming language perspective, they need to just finish the functionality.
> It is a language, better than MatLab/R/SPSS, which come before it.
I don't think so. I think R is a lot more expressive and not really any harder to read. It might have a steeper learning curve, but it's not so bad that I think that actually matters.
I'm currently learning R. Its a terrible, dreadful programming language. But an excellent DSL for statistical analysis. The main mechanisms for the expressive nature of R is its use of the Environment structure, and terrifying casual use of call stack inspection just to make symbolic APIs. It doesn't even make the latter part of the language without Tidyverse things like rlang. In fact without the Tidyverse efforts the language is even bad as a DSL unless you only deal with a CSV of data with like 100 data points.
I think R is a much worse programming language, like 1 indexing, very unintuitive string manipulation, dataframe being the magic facade that hides complexity, etc.
I would choose Python any day if the other option is R.
1-indexing makes sense within the realm in which R shines. It’s only CS people who seem to completely lose their shit when they encounter it. I’m glad Julia also has 1-indexing.
If you are a computer scientist doing a little data science, then R is awful for the reasons you detailed. If your function entirely consists of data science, then R is a fantastic language for the same reasons you hate it.
R is a fantastic statistical toolkit and a terrible programming language. Building software in it? Massive pain in the ass. Processing tabular datasets on the other hand is an absolute breeze that leaves pandas in the dust. Tidy verse makes it better though.
Yeah, this is generally my take. On the one hand, I cringe when I see all the discussion regarding how much better Python is than R for data science. As a data scientist, who spends 90% of my time cleaning, manipulating, and modeling data, I would take R any day over Python. But at the same time I realize that for many, data science is more 90% software engineering, with 10% being the actual data manipulation and modeling, and for these people, R is a complete non-starter.
I look at developer surveys sometimes when I'm trying to decide what to learn next. According to the 2018 stack overflow survey, 68% of python users would like to use it again in the future [1].
The surveys never tell me why though. What do people like or dislike about python? I know it has a lot of libraries people love (scikit-learn, tensorflow come to mind)
Those surveys are often targeting mosstly newer devs who know no better because they did not have the time to validate their opinion against real world.
Its not thir fault tho, you can only do so much in limited time, thats why expertize requires years.
I would rather wish Golang eat the world than Python, just because the practicality of python becomes questionable when performance is key.
In my previous startup in India, I trained unskilled personnel to become decent python developers to work on our product; everything was fine till the product grew exponentially and we had to optimise every nook and corner of it to better serve the customers and save on server bills.
So we had to optimise our application with Cython to meet the demands. So, when training someone to code if we use Python as a language; we should follow up with the disclaimer "You will learn to code faster than most other languages, but you cannot use this knowledge to build something at scale (i.e at-least when budget is of concern, when you are not instagram)".
In comparison, Golang excels in both form and function. It is just as easy to teach/learn as python and doesn't need the aforementioned disclaimer. Web services are a breeze to write and is built for scale.
I understand that there are criticisms against the language design of Go, some are valid and most are just because it was made by Google but none questioning the scalability of Go applications.
Many of my complaints about Python are valid for Go, except that Go makes even more perilous decisions than Python for error handling (and the community kinda gleefully embraces it).
But at least Go is a lot faster and has real concurrency AND parallelism, so it's definitely better than Python.
Go makes worse decisions even, I'd say. Python has gevent, one of the best and most friendly greenlet libs around too, so the concurrency/parallel issues seem fairly moot. It's true that the batteries-included versions of things, while mostly easy to use, falter at scale.
> so the concurrency/parallel issues seem fairly moot.
Python has the GIL, so true parallelism can only be achieved with basically cloning a process X times the number of processors on a computer and inter process communication.
I’m really glad to hear someone else voice these frustrations. I’ve really tried to embrace Python and everyone tells me how great it is. I feel like a failure as a developer but I dislike it so much I quit a job, that switched to Python as the primary language, and took a year off from development. I’m pretty much a polyglot as far as languages go but Python riles me.
Python is the simplest mainstream language yet still reasonably powerful. Some folks don’t like simple and I’ve found it’s better in the long term to find those that do.
>that infernal pep-8 linter insisting 80 characters is a sensible standard in 2019
So change the limit or disable that check. If someone is keeping you from doing that they're the one who's insisting on 80 characters, not the language. Who uses any linter without making some changes to its default settings?
I was smart and lucky enough to switch from Python to Clojure and from Javascript to Clojurescript. I am not even sure anymore what have I liked about Python back then. I know for sure - those who really like Python probably haven't given a heartfelt try to Clojure, Haskell or Racket.
(and that infernal pep-8 linter insisting 80 characters is a sensible standard in 2019) increases my burnout by 100x
I've yet to encounter a python linter where you can't pick and choose which rules to ignore. This is a first one to go. Annoying PEP for sure, but https://pypi.org/project/black/ almost completely eliminates your issue.
There is a lot of hype and "they are using it too" mentality as well I think. I am working on control software which, if it crashes or even takes a tiny bit too long to issue a command, could cause the company big financial loss. I was forced to do it in python "because everyone else is using python".
This is about as sensible to me as saying it is a good idea because most trains are longer than they are wide. Who cares about those things, we use computers that nearly all have 4:3 or 16:~10 displays (or wider), and many of use use more than one of them.
I obviously don't agree, but it is worth noting that comparing typographic conventions for English with typographic conventions for code is not a very good idea, to me. Especially when we're discussing a language with semantically active whitespace.
A 100 character line might only see 15-70 characters of active use.
While this is a fair point, pylint bumps the limit to something like 100 characters, and you can split deeply nested logic into separate functions with ease.
Are you implying short and wide trains would be sensible? I like using a wide screen for programming, but prefer code formatted to 80 chars. That allows me to have two vertical windows of code open side by side. It also makes it easier to use things like a graphical diff to merge code.
it's funny because a few people ended up using python as a prototyping tool to find better approaches to then rewrite in cpp or else. Somehow in that regard it saved them tons of time.
You can configure Pep8/Flake8 to ignore subset of rules. You might wanna look into that. Also, automatically executing it might be your preference, or not.
What I find worse about Python than other languages is the lack of tooling and a relatively small collection of libs - many of which are half way done. In a last assignment, we decided Python is at most a hobby language. Python is great for machine learning because most research was conducting using this language, and as such, tooling is available. I would use it at most as an API exposing ML but that's pretty much it.
Well, that's what I said, there are plenty of packages for data science. For building APIs and web stuff usually there aren't. Python is nowhere near NodeJS for instance. But whatever suits, if you only know one language, it will seem like the best language out there.
This comment is so strangely off the mark I almost wonder if you're basing this off of some secondhand hearsay. Until recently, Python's largest arena of usage was web development and API scaffolding by far. A simple search would have revealed literally thousands of web development focused libraries, most centered around the Django ecosystem [1].
As another commenter mentioned: Django, DRF, Flask
But unmentioned... The old titans of Pyramid, Pylons, CherryPy, Bottle, Tornado, wheezy, web2py, and more.
A _gigantic_ portion of the community is centered around web development, and the fact that almost all web packages have centralized around Django, DRF, and Flask is a function of dozens upon dozens of popular frameworks merging in a "survival of the fittest" fashion into the best possible union of ideas. You seem to perceive "many packages" that all perform a similar function to be a good thing, but speaking from the perspective of someone who writes 70% Javascript and 30% Python I'll tell you that almost every other sane group of developers considers the highly duplicitous JS ecosystem to be a massive weakness.
The NPM ecosystem has an _ocean_ of shallow, highly similar, and one-off frameworks because Javascript developers tend to "divide and conquer" rather than bring good ideas together. Python developers deprecate their packages in favor of better ideas, or just open a pull request to alter the project rather than creating a 99th competing framework in the first place.
You have Django and DRF and Flask. DRF is, by far, the best designed framework for building APIs in any programming language I have ever seen, and I've at this game for a long time.
JavaScript has 10 times more packages than PyPi, and for the most part they are absolute garbage with no intent on being maintained.
Python has substantially fewer packages where the community rallies around them and keeps them as best-in-class. SQLAlchemy, DRF, factoryboy, requests, etc., are all incredible one-stop-shops for the vast majority of use cases.
You don't need 15 libraries for doing HTTP requests. You just need one good one that does the job so you never have to think about the problem ever again. Python excels at this class of libraries and by comparison npm has appalling choices.
I'll take a framework that has worked effectively for a decade without significant changes where I can use all my accumulated experience to get things done over the random musings and experimentation of the half-baked JS libraries where you're stuck for hours solving relatively trivial issues that I have never had to think about in DRF because DRF just works and is backed by some of the most experienced people in API architecture in the industry.
When you think about and iterate judiciously in a problem domain over a decade instead of trying something different every day to see what sticks, you'll see you can get remarkably far.
And what exactly are Flask and Django missing to merit more packages doing the same thing?
Routing requests and managing HTTP fundamentals is a solved problem. There is literally zero value in adding another framework when the real complexity is in business logic.
That’s the issue. These are routing requests and not much more. I recommend looking around at java, node, php, c# web frameworks for more details on what a web framework should do. Sqlalchemy and other hobby libs can extend flask and django but those are similarly limited.
> In a last assignment, we decided Python is at most a hobby language.
I don't know if you are just trolling, but this is the silliest, most detached from reality thing I've read online today. Python was key in building numerous massive public companies like Instagram and Dropbox. It's one of the most popular and widely used programming languages on the planet for everything from APIs to desktop clients to data pipelines. It had a lot of early popularity in the Silicon Valley start-up scene in the early 2000s, even pre-dating the Ruby on Rails web dev trend.
The guy who founded the company that runs this very website wrote about the draw of Python 15 years ago [1] at which point it was already widely used in certain niches. This is before any deep learning libraries existed. I remember first playing with Python around 1999 or so.
> I would use it at most as an API exposing ML but that's pretty much it.
I don't know how old you are or how long you've been in the industry, but the ML thing is a "second act" for Python. Deep learning grew up in a time and place where Python was a good fit which put Python in the right place to benefit. But Python had already lived a long and full life before any of that happened.
It's fine if you don't like Python or don't think it is a good fit for a project, but claiming it is a "hobby language" with a "lack of tooling and a relatively small collection of libs" is a good way to get laughed out of the room. It has one of the largest and most diverse libraries. And as far as tooling, Python is one of the most popular languages for implementing tooling. Check out ansible.
> There are plenty of packages for data science. For building APIs and web stuff usually there aren't. Python is nowhere near NodeJS for instance.
This has got to be a troll, right? Just some of the most popular web frameworks: Django, TurboGears, web2py, Pylons, Zope, Bottle, CherryPy, Flask, Hug, Pyramid, Sanic... Lots of huge websites were originally built with Python like Instagram, Reddit and YouTube. Of course they mature into complex distributed architectures using all kinds of languages, but it's all about the right tool for the job.
It appears tho the right tool for the task at hand, as a codebase grows, is NOT Python. A good starting point for someone fresh of off a university campus, and easily impressionable. An "object oriented" language without even the basics of scope visibility is not object oriented. Half done libs are not libs that one might consider production grade. Debugging tools where you need to dive into execution branches and heck knows what other archaic debugging techniques are not debugging tools. Lacking documentation or poorly written readme fiels are not documentation. This whole Python movement is gaining a bit of traction because it became popular in universities. The realities of real life programming will sway many religious programmers towards more suitable languages.
So yeah, fantasising about how Python "is eating the world" is a good dream, but the only thing that Python is eating is dust left behind by far more developed programming languages, surrounded by far more modern ecosystems around them.
If only its package management were as easy as its syntax...
I wish pip worked the same way as npm: -g flag installs it globally, otherwise it creates a local "python_modules" folder I can delete at any time. And optionally I can save the dependency versioning info to some package.json...
Instead, pip is a nightmarish experience where it fails half the time and I have no idea where anything is being installed to and I'm not sure if I'm supposed to use sudo or not and I'm not sure if I'm supposed to use pip or pip3, etc.
2. For each project you want to work on, create a venv. Yes, there are tools for this, but the base venv tool is totally fine. (venv should be included in your Python, but a few distributors like Debian put it in a separate package - install it from them if needed.) Use python3 -m venv ~/some/directory to create a venv. From here on out
3. As a first step, upgrade pip: ~/some/directory/bin/pip install -U pip.
4. Install things with ~/some/directory/bin/pip install.
5. Run Python with ~/some/directory/bin/python.
Slightly advanced move: make a requirements.txt file (you can use .../bin/pip freeze as a starting point) and use .../pip install -r requirements.txt. That way, if you get any sort of package resolution error, you can just delete your venv and make a new one. (Downloads are cached, so this isn't super annoying to do.)
A "project" can either be actual Python development, or just a place to install some Python programs and run them out of the resulting bin/.
(Edit: Yes, I know about activate, see the replies below for why I don't recommend it. With these rules, you get to say "Never ever type pip, only .../bin/pip", which is a good safety measure.)
Herein lies my problem. If I want to start a Node project I run `npm init` and then `npm install --save` to my heart's content. If I somehow manage to mess it up I just delete node_modules/ and install again.
If I want to start a Python project I have to set up venv and remember to put relative paths in front of every command or else it'll run the system version. Sounds simple, but it's still something to always remember.
1. pip install --user and sudo pip install are fine actually, they will not interfere with venv, they can co-exist just fine.
2. yes
3. probably do "source bin/activate" first, then run 'pip install -U pip'
4. just run pip install whatever, no need the full PATH
5. just run python directly, no need the full PATH
6. run 'deactivate' when you're done for now, 'source bin/activate' when you want to continue/resume sometime later
in fact I like this better than node_modules, the venv layout of bin/include/lib is more natural comparing to the odd name of "node_modules" in my opinion, and I don't need npx etc to run commands under node_modules either, all are taken care by venv with its 'activate' script
pip install --user and sudo pip install won't break your venv. But they will break your system Python and any OS commands that depend upon system Python, perhaps including pip and virtualenv themselves, which is incredibly confusing. I've helped both friends and coworkers un-break it, and the symptoms aren't generally obvious. I wrote the patch to pip in Debian to prevent sudo pip install from removing files from Debian packages via upgrading packages. It's a huge pain, it's only worth running if you know exactly what you're doing, and as someone who does know exactly what they're doing I can attest that it's never necessary. After all, you can always just make a virtualenv.
One thing I did at my last job was to make a Nagios alert for machines with files in /usr/local/lib/pythonX.Y/site-packages, indicating that someone had run a sudo pip install, which was super helpful for "why is this machine behaving slightly differently from this other machine which should be identical". We had a supported workflow involving virtualenvs and also we had multiple members of the Debian Python team on staff if you needed systemwide packages, so not only was there always a better solution, there were people to help you with that better solution. :)
Re activate/deactivate, that's a matter of taste but I find it easier to avoid it completely too - see my reply in https://news.ycombinator.com/item?id=20672299 for why. Basically, you get the simple rule of "Never run bare pip" instead of "Remember which pip is your current pip and whether it's the one you meant."
> But they will break your system Python and any OS commands that depend upon system Python
Sudo pip install might on some distros (and I consider this to be a bug on the distro level, not a Python issue) but I've never heard of --user breaking anything
Maybe I'm misremembering, but, isn't the point of pip install --user to get things onto your import path when running the base Python interpreter, just like sudo pip install would (except scoped to your user)? If so, wouldn't installing an incompatible newer version of some library (or worse, a broken version) break system commands that import that library, when that same user is running those commands?
I've grown to dislike activate, because it breaks the simple rule of "never run pip, python, etc., only run your-venv/bin/pip, python, etc.". Now the rule is "Don't run pip, python, etc., unless you've previously run activate and not deactivate" - and it has the complicated special case of "make sure the command exists in your virtualenv." (For instance, it's definitely possible to have a Python 2 virtualenv where pip exists but not pip3, and now if you run pip3 install from "inside" your virtualenv it's global pip! Or you might have a Python 3.6 virtualenv and type python3.7 and wonder where your packages went, or several other scenarios.)
If you have the shell prompt integration to remind you whether you're in a virtualenv or not, it's fine, but I don't always have it, and I find it helpful to manually type out the full directory name (generally with the help of up-arrow or tab...) so I know exactly what I'm running.
for bash/fish it automatically prefix with a (your-venv-name) so it's obvious you're under some venv, not sure about csh but I would assume it will do something similar. looks like venv supports bash/csh/fish only by default however.
I agree. I'm optimistic about tools like Poetry https://poetry.eustace.io/docs/basic-usage/ for solving this. Unfortunately Python predates the realization that good packaging tools were a thing that needs to be solved in the core language and not externally (Go, Rust, Node, etc. postdate this realization; C, Perl, Java, etc. also predate it).
The flip side is that decoupling the interpreter/compiler from the build system makes it more possible to write tools like Poetry (and, indeed, virtualenv) that explore new approaches to the problem. At my day job where we do hermetic in-house builds with no internet access and (ideally) everything built from checked-in sources, building C and Java and Python is straightforward, because we can just tell them to use our source tree for dependency discovery and nothing else, and we can set up CFLAGS / CLASSPATH / PYTHONPATH / etc. as needed. Building Go and Rust and Node is much more challenging, because idiomatic use of those languages requires using their build tools, which often want to do things like copy all their dependencies to a subdirectory or grab resources from the internet.
Of course, given that it's Python, there should be one - and preferably only one - obvious way to do it....
>Unfortunately Python predates the realization that good packaging tools were a thing that needs to be solved in the core language and not externally (Go, Rust, Node, etc. postdate this realization; C, Perl, Java, etc. also predate it).
Sure, but it's also a cultural thing. Ruby is nearly as old, and also predates this, but has nowhere near the insanity of Python. The community jumped on bundler and rvm/rbenv/etc super quickly, and rapidly improved them, while the Python community is barely even aware of pip-tools / pipenv AFAICT. Even virtualenv is really only a "real pythoners know" thing, it's rarely mentioned in guides, so newbies screw up their global environment frequently.
Ruby was exotic until someone translated the docs to English, but the whole ecosystem is indeed one reason I love Ruby. I really don't understand why python 2.7 is still a thing 11 years later. Sure, legacy systems, but if I install recently active open source on my machine I wouldn't expect it to use an outdated version of a programming language. Upgrading can't be that hard.
python 2.7 is still a thing because it was announced as the LTS* release over a decade ago.
* nobody called it that, but thats effectively what it meant to say “there will be no python 2.8. Python 2.7 will be supported until T. Python 3.x will be superceded in a year.”
Python 3 comes with it as "python -m venv". Once in the virtualenv, you don't have to worry about the various pip forms and effects, you can just pio install.
You can get fancier than that of course, but that's what works with stock python, on all OS.
I haven't seriously JavaScript'd in a couple of years but my problem with it then was different versions of node or npm. Nice thing about python virtual environments is that problem never exists (can make environments with whatever version you want).
pipenv really solves the Python version problem, IIRC. I don't actually use pipenv, myself, since I haven't had time to thoroughly figure out the new magic and I prefer to know exactly what's going on with my Python's.
npm doesn't solve the duplicating of deps any better than python/pip, as far as I know. The react-native app I'm currently working on has a node_modules folder at 960Mb. That's probably bigger than nearly every virtualenv I've ever seen. A react-native node_modules on a bare project with `--template typescript` is at least 350Mb (created one a few minutes ago). I'm using nvm for node version management. No problems so far.
Exactly. NPM gets a lot of hate but lockfiles and local install by default is great. The default mode should not be global installation. Also imo virtual environments aren't amazing. Having some mode you have to remember to flip on that changes essentially the semantics of your pip commands seems a little brittle. Tools that work on top of pip and venv like Pipenv or Poetry seem a lot better.
This isn't even the start of the problems with pip and pypi. If I install pylibtiff, it embeds a copy of an old libtiff which is binary incompatible with the libraries on my system. I have to build it from source, then it works just fine. But I can't inflict this level of brokenness on my end users.
This applies to many packages embedding C libraries, including numpy, hdf5 and all the rest. There has been minimal thought put into binary compatibility here, meaning that it's a lottery if it works today, or will break in the future.
I couldn't agree more with this. I was forced into doing some UI coding and although I could never full embrace js, the package management aspects (esp having the sane default of not installing packages globally) were definitely superior to python.
I feel uncomfortable with the fact that people feel a third-party solution is the best way to solve this mess. It can also get messy when packages installed with pip, pip3, conda and apt are all entangled with one another in various ways.
It’s unfortunate that it’s third party, but conda has the unquestionable advantage of being the only Python-centric packaging system that has a reasonable shared binary library story
I'm curious, do you not find wheels + manylinux reasonable? I agree that until recently, Conda definitely had that advantage, but now that you can `pip install scipy` and have that get you a working library and not try to compile things on your machine what does Conda offer beyond that?
I guess one thing Conda has that the pip ecosystem doesn't is that it supports installing non-Python shared libraries like libcurl on their own. Is that an advantage? (We absolutely could replicate that in the pip ecosystem if that was worth doing, and it's even not totally unprecedented to have non-Python binaries on PyPI.)
I think it would definitely be great if pip could install non-python dependencies. One problem right now is that many projects will tell you to just pip install xyz. You execute that, things start building, and the process fails partway with some cryptic message because you're missing an external dependency. You figure out which one, you install it, start again, and another dependency is missing. Rinse and repeat. It's definitely not a turnkey solution, and this issue trips up newcomers all the time.
With respect to versioning, I think pip should be way more strict. It should force you to freeze dependency versions before uploading to pipy, not accept "libxyz > 3.5", but require a fixed range or single version. That would make packages much less likely to break later because newer versions of their dependencies don't work the same way anymore.
Does pip allow version number dependencies? Conda is able to upgrade/downgrade packages to resolve conflicts, whereas pip just seems to check if a package exists and shrugs when there's a version conflict.
pip does handle versioned dependencies and ranges, and know enough to upgrade existing packages when needed to resolve an upgrade. Its resolver isn't currently as complete as Conda's - see https://github.com/pypa/pip/issues/988 . (On the other hand, the fact that Conda uses a real constraint solver has apparently been causing trouble at my day job when it gets stuck exploring some area of the solution space and doesn't install your packages.... so for both pip and conda you're probably better off not relying too hard on dependency resolution and specifying the versions of as many things as you can.)
... the same thing happens if you mix stuff in your usr/bin directory that isn't managed by your system package manager.
The solution is: don't mix your package environments. Use a conda environment. Just like in Linux, you'd use a container. If you wait for the Python steering committee to fix pip you'll be waiting a long time.
Yes, exactly my point: then you have to potentially deal with conflicting dependencies between pip and conda packages. This happens and it's a pain to deal with.
Holy Crap! What a lot of irrational, hyperbolic hate for Python.
I think everybody should spend their first couple of years working in Fortran IV on IBM TSO/ISPF. No dependency management because you had to write everything yourself. Or maybe [edit: early 90's] C or C++ development where dependency management meant getting packages off a Usenet archive, uudecoding and compiling them yourself after tweaking the configure script.
I'm not saying Python is perfect, but if it's causing your burnout/destroying your love of programming/ruining software development you seriously need some perspective.
I just returned to Python for the first time in a little while to collaborate on a side project and ran into a few tricky-to-debug errors that caused a fair bit of lost time. Know what the errors were?
In one case, I was iterating over the contents of what was supposed to be a list, but in some rare circumstances could instead be a string. Instead of throwing a type error, Python happily went along with it, and iterated over each character in the string individually. This threw a wrench into the works because the list being iterated over was patterns, and when you apply a single character as a pattern you of course match much more than you're expecting.
And another one, I typoed a variable name as groups_keys instead of group_keys (or vice-versa, I don't remember). Instead of just throwing an error, Python happily went along with it, used an uninitialized value, and then all the logic broke.
There's entire classes of errors you can cause yourself in Python that aren't possible in stronger, statically-typed languages. For a large project, I'd pick the old and boring Java over Python every time.
Python is a dynamic language, that's what dynamic languages do, you don't have a type checker but have greater flexibility, but you don't have to settle on that, you can actually use mypy and annotate types and get best out of both worlds.
> And another one, I typoed a variable name as groups_keys instead of group_keys (or vice-versa, I don't remember). Instead of just throwing an error, Python happily went along with it, used an uninitialized value, and then all the logic broke.
This isn't what Python would do, if the variable was undefined Python will throw an error, so you must have defined it with this name or you're misremembering what have happened.
It has nothing to do with static vs. dynamic. There's no reason that being an early-binding language that a string has to be iterable itself, and the proposal to change this was only rejected as it broke too many things[1] and couldn't be automatically fixed.
Point in the GP's favor: Fixing it would definitely not be a problem with an early-binding language! In fact, the nigh-impossibility of automated refactoring puts lie to the notion that late-binding languages are more "agile."
It's a design flaw, in the same way Python 2's allowing comparisons between different types was a flaw, e.g. "a" < 3 succeeds. Python 3 now, correctly, throws a TypeError because there's no sensible ordering between the two things.
(While I'm griping: Another design flaw is conflating iterables and iterators, which makes generators almost useless. Say a generator is pass to a function expecting an iterable. If the function uses it twice, the second time it silently returns nothing!)
> This isn't what Python would do, if the variable was undefined Python will throw an error
I think GP must have assigned to the name, in which case Python will create a lexically bound name.
Python's rules for naming can make perfect sense or be quite surprising:
try:
x = four()
except Thing:
x = 5
print(x) # 4 or 5
for a in [1, 2, 3]:
pass
print(a) # 3 ?!
mypy is a great effort, but very experimental. Try using it on any real-world large enough project and it loses most of its value as there are still a lot of unimplemented things, or because you'll depend on a third party module that hasn't support for it yet.
Case in point: Pandas, the foundation of data programming in Python, does not provide the Series or DataFrame (that's a table) types in a way that MyPy can use.
Your 2nd error isn't possible in Python, so I'm not sure what you did there. Regarding the first, sure, it is a bug that was annoying to catch. But, having an `Iterable` interface in Python is also really neat and useful if used responsibly. If you're programming regularly in Python, you are accustomed to the tradeoffs that come with a dynamic programming language and no static types, and you can still avoid issues like the one above.
Right off the top of my head, using type hints with a decent IDE or an assert statement would likely have caught the issue.
I'm not saying that Python doesn't have issues (all languages do), but I don't see the error noted above as any sort of deal breaker. On the other hand, if you're only ever going to use Python like a strongly typed language without taking any advantage of its dynamic characteristics, then I can see why it would seem as a total downgrade compared to languages like Java.
I didn't explain the second one well. Here's some exact code.
group_keys = ...
if not isinstance(group_keys, list):
groups_keys = [ group_keys ]
So rather than listifying the non-list variable, it was creating a new variable. The cause of this bug is that Python doesn't distinguish between declaring new variables and overwriting existing ones.
Well, this should have been caught as an unused assignment in static analysis. A whole ton of languages allow this situation, so I'm not gonna ding Python too hard for that one.
However, here's a related but different python gotcha:
if foo(a):
v = list(bar(a))
for i in v:
print i
In this example, v is only defined inside the if. Due to python's limited scopes, v is also valid outside the if, but only has an assignment when foo(a) is True. When foo(a) is false, the for loop throws an NameError. And yes, a coworker wrote code that accidentally implemented this antipattern, albeit much more spread out in the code base.
This is clearly a bug in the code, yet no static analysis tools I've tried have successfully discovered it. There's a bug in pylint that's been marked WONTFIX because it requires a wildly different implementation. At a language level, it feels weird that if blocks aren't a scope level for new variables. If you want to reference v outside the if loop, declare / assign it outside the loop first.
statically avoids this problem. In general, prefer immutable variables where possible. Single-assignment form is nice for a lot of reasons, not the least of which is that it avoids this particular gotcha.
And I should add that the "right" way to do this would be to factor this out to a function:
group_keys = coerce_to_list(...)
is much clearer than either block, and avoids the possibility of the issue.
All of these things are true, but they require a non-trivial level of experience and discipline to avoid most potential gotchas. Your average Python project on the Web isn't written to this level of quality, and when people are learning programming using Python in school they certainly aren't there yet, and are gonna hit all kinds of problems related to this stuff.
But is there a way to force immutable variables in Python? You can easily still end up in the same situation when you typo something (easy to do when plurals are involved), and then end up reassigning something when you meant to create a new variable.
I don't think that's fair to be honest. If you had simply used Pycharm with default settings you would have easily caught the first bug due to the linting. It's a fair complaint, but this specific bug is easy to catch using any modern Python IDE.
I've never found the "Use this specific IDE" defense particularly valid, considering that many IDEs don't have these features and that in other languages the compiler itself protects you.
Needless to say, I was not using Pycharm for this development, nor am I likely to install an entire IDE just for a small change I'm making on a random project. It's a non-trivial burden to configure and learn an entire IDE, vs just using what I already know (which is often just emacs).
It's even harder to take "The IDE should make up for deficiencies in the language" seriously. In languages that handle this stuff well, you can edit in Notepad and still not make these mistakes. Why push it up several levels to a few specific IDEs that most people don't even use?
> But is there a way to force immutable variables in Python? You can easily still end up in the same situation when you typo something (easy to do when plurals are involved), and then end up reassigning something when you meant to create a new variable.
Not always. Mypy has experimental support for `Final[T]` [0], and attrs/dataclasses support final/frozen instances, but that's opt in on a per-argument basis.
I see this often and it is a bad pattern that people do.
Typically type checked languages wouldn't even allow you to do. If you would use mypy for type checking it wouldn't like it because you're redefining the type of a variable. Best practices would suggest you use a different variable for the conversion if you must, but ideally you should just make the function accept list as an argument. If you're really worried about passing something else than a list, you should use type annotations to tell type checker what it is. If you want to add extra runtime check then do:
assert isinstance(group_keys, list)
You can complain that Python allowed you to something dangerous, but you have tools to help you avoid it and this flexibility is what makes tools like SQLAlchemy so powerful.
I still don't think you quite understand what's going on here. Python wouldn't create a new variable in this case. I It would re-assign the value represented by the variable you already assigned once. I agree that it would have been better if Python had explicit variable declarations (this is one of the few things I think Perl got right.).
On the other hand, Ruby made this same mistake. If you wrote this code in Javascript you wouldn't get an error, but you would in fact have two different variables.
For instance, this code runs for me using node 8:
var fun = (bool) => {
var x = 1;
if (bool) {
var x = 2;
x += 1;
} else {
x += 1;
}
console.log("x=" + x);
}
fun(1);
fun(0);
> I still don't think you quite understand what's going on here. Python wouldn't create a new variable in this case. I It would re-assign the value represented by the variable you already assigned once.
Uh. no. he typoed the reassignment, so it wouldn't re-assign the value.
> So, of three of the most popular dynamic languages, Python, Ruby, and Javascript, none of them would have helped you catch this kind of error at script-parsing time. So again, it seems like you have an irrational dislike for Python, all things considered.
Sure, but he's made it clear he likes Java. Fundamentally he's against dynamic typing, so of course he doesn't like any of the dynamic languages.
I don't understand why you're accusing me of being irrational. These seem like very rational problems to have with Python. They literally caused me bugs that cost me time to deal with that I wouldn't have faced in other languages.
You're also assuming that I don't have the same problems with Ruby or JavaScript. I do. The exact same critique could be made of them as well, but they're not the subject of this thread; Python is.
You can't argue with someone who has chosen to overlook your viewpoint.
I've ran into the same issues while writing python code. People who are newly picking up python are especially prone to these kind of bugs. Also, with python I have to spend lot of time to figure out what went wrong in my code as compared to other languages.
People who have been using python for long have wired there brain to avoid such pitfalls and now they happily defend it.
I don't think what you're saying is true. I already said I think it would have been better if Python and Ruby had explicit variable declarations. But, if this is your biggest issue with a language and it's ecosystem, then IMO that language is doing pretty well. I would rather, for instance, have to deal with implicit variable declarations in Python that the gigantic mess of Java frameworks that have been invented to "reduce boilerplate", such as Spring/Guice, AspectJ, Hibernate, etc.
My bad. I didn't realize you were against all dynamic languages in particular. FWIW I prefer Java and static types as well, but as far as scripting languages go, I think Python is pretty great.
I disbelieve. And I disbelieve despite being a fan of dynamic languages.
The tradeoff is that dynamic languages are faster to develop, more concise, but more expensive in maintenance exactly because of issues like this. The data that I base this opinion on is an unpublished internal report from nearly a decade ago at Google quantifying costs of projects of different size in their different languages. Which was Java, C++, and Python. Python had the lowest initial development, and the highest maintenance costs. That is why Google then moved to Go as a replacement for Python. It was good for the same things that they used Python for, but being statically typed, its maintenance costs were lower.
I can believe that. But for a lot of people, the lower initial development time/cost aspect matters a lot. If I had Google resources, sure, I'd Go with other languages perhaps, but you can still write high-quality and capable software in Python. And while the batteries included aspect of Python is not everyone's cup of tea, I personally find it quite handy to have that so I don't have to waste a ton of time evaluating different libs to do fairly standard things.
To be clear - I'm not trying to say that Python is better in any objective way. Ultimately, I think people should use the tools they have available and prefer, to build what they want.
But for a lot of people, the lower initial development time/cost aspect matters a lot.
As I said, I'm a fan of dynamic languages. :-)
One of the top ways that startups fail is failing to build what they need to build quickly enough. Maintenance costs only matter if you succeed in the first place. Using dynamic languages is therefore a good fit.
But, even if you're not Google, if you're writing software and have the luxury of paying attention to the lifetime costs of the project up front, you should choose a statically typed language.
That would not catch the bug if the input is not under is control.
You could as well say "Just check if the object is a string" in the method, which would work but the point was rather that it is difficult to notice if you did not think about it. Compared to other languages that would crash or not compile instead.
Yeah, the input isn't really under control because it's coming from deserializing a YAML file. It worked for the exact type of input I was expecting, namely, when you configure a specific value as a list, but it wasn't working for anything else. And YAML has plenty of types it can split out, so my naive fix still only handled lists and strings properly!
Yeah, YAML deserialization is the worst case scenario for dynamic typing. In most situations, types are pretty consistent and assuming you run your code at least once, you'll find most errors. But with YAML deserialization all bets are off. YAML is even worse then JSON for this because seemingly minor changes in the YAML can change the shape of the data.
I've had success validating such data against a schema, so I know it had consistent type structure before working with it.
>Iterable` interface in Python is also really neat and useful if used responsibly.
Honestly this was a major attraction to python for me a decade plus ago as a student when I started learning--even when I used it irresponsibly. There are so many small tasks where you just kinda have to iterate over 100-1000 items that you're not worried about big-O or anything like that—you just want to iterate and work on a collection quickly for some task in the office.
>In one case, I was iterating over the contents of what was supposed to be a list, but in some rare circumstances could instead be a string. Instead of throwing a type error, Python happily went along with it, and iterated over each character in the string individually.
I've been using python for about 13 years professionally and I wrote up a list of "things I wish python would fix but I think probably never will" and treating strings as iterable lists of characters was on there.
I've seen this bug multiple times and the fix is relatively easy - just to make strings by default non-iterable and use "string.chars" (or something) if you really want to iterate through the chars.
Nonetheless, I still love the language and wouldn't use anything else.
>Instead of just throwing an error, Python happily went along with it, used an uninitialized value, and then all the logic broke.
This one gets caught by linters - unfortunately 90% of most python linters spit out rule violations which aren't important which drowns out stuff like this in the noise.
* Implicitly casting strings, integers, dates, etc. to boolean (e.g. "if x" being true if x is a non empty string). Cause of more unexpected bugs that I can count, but would cause massive headaches if implemented and memories of the the 2-to-3 transition would scare anybody away from doing this I think.
* Treating booleans as integers (True + True = 2). Probably wouldn't cause that many headaches if implemented but everybody still seems to think it's a neat idea for some reason.
* Treating non-package dependencies of pip packages (e.g. C compilers, header files) as something that is either the package's problem or the OS's problem. Nobody looks at this problem and thinks "I should solve this".
Iterating over characters in a string is something that's done very often in introductory CS classes, but very little in the real world. Python has support for string finding and regexes; why in the world would I be individually iterating over characters? Generally, when you see that, it's a code smell.
So yeah, I totally agree with you, it'd be better if trying to iterate over a string were a flat-out error, and if you really want it, you should mean it. Though Python being dynamic still means that you'll only spot this error at runtime.
As for linters, how do they know if your intent was to reassign the value of an existing variable, or to define a new one? The language has no way to indicate which of these is intended.
For your first error, you can do some foot-shooting with a statically typed language too.
I remember a bug I made using C#, where I wanted to delete a collection of directories recursively. I got mixed up into the various inner loops and ended up iterating over the characters like you. But C# allows implicit conversions of char to string so the compiler was OK with it, and since those where network drive directories (starting with "\\server\"), the first iteration started deleting recursively the directory "\", which in windows means the root directory of the active drive (c:\)... And SSDs are fast at deleting stuff.
> And another one, I typoed a variable name as groups_keys instead of group_keys (or vice-versa, I don't remember). Instead of just throwing an error, Python happily went along with it, used an uninitialized value, and then all the logic broke.
Python doesn't have uninitialized values, it throws NameError when you try to access a variable that hasn't been set. So I don't see how this could have happened.
Well this is anything but a new complaint. I would assume a user who has worked in Python for some modest amount of time to have made peace with this. One works in Python knowing that this can and will happen (well one does have linter on steroid like mypy now to counter these).
Python code needs more testing, more run time type checking of function arguments than a statically typed language. If that's a deal-breaker then one shouldn't be using Python in the first place. What you gain though is some instant gratification, and the ability to get something off the ground quickly without spending time placating the type checker. Its great where your workflow involves lot of prototyping, exploration of the solution space and interactive use (ML comes to mind, but even there int32 vs int64 can byte, correction, bite). I see it as a trade off -- deferring one kind of work (ensuring type safety) over another. Hopefully that deferral is not forever. I like my type safety but sometimes I want that later.
What I typically do is once I am happy with a module and I do not need the extreme form of dynamism that Python offers (something that's frequently true) I take away that dynamism by compiling those parts with Cython.
> In one case, I was iterating over the contents of what was supposed to be a list, but in some rare circumstances could instead be a string.
The creator of a well-known alternative to Python has a single-letter email address, and regularly receives email generated by Python scripts with this exact bug (which means instead of sending an email to "user", sends an email to "u", "s", "e", and "r"). So I’ve heard.
In my CS program, we learned Python as a convenient way to sketch a program. We also learned C++ for speed and OCAML for those functional feels. A programming language is a tool, Python has some great use cases mostly focused around ease-of-programming.
The bugs you describe should both be easy to catch with unit tests. It sounds like the problem is not that you're using Python, it's that your project lacks tests. Sure, you can typo this sort of thing; but it should be apparent within seconds when your tests go red.
(And nowadays, you can also use type hints to give you a warning for this kind of thing, e.g. your IDE/mypy will complain about passing a string where the function signature specified a List.)
Serious question: If you are writting unit tests to check types, why not just use a language that has a compiler that does that for you? And if you are writing python with type hints, why not just use a language that uses the types you spend time adding to make your program faster.
Python is great for sharing ideas / concepts, but under some circumstances it seems irresponsible to choose it over other viable options like Go (if you use Python because it's easy), or C# (If you use Python because it's a 'safe' enterprise choice). (Ecosystem specific things aside at least)
As the sibling comment said, I'm not proposing checking types in unit tests, I'm proposing checking that the behaviour is correct.
If there's a code path that passes in a bare string instead of a list, and your logic breaks, then that code path should have a failing test case. However, type hints can provide another opportunity to catch this kind of mismatch before they even get committed.
> under some circumstances it seems irresponsible to choose it over other viable options like Go (if you use Python because it's easy)
This is probably true, but I think people tend to overuse this argument (i.e. use an overly broad set of "some circumstances"). I build fintech apps with Python, for example, and don't find any of these issues to be a
problem. In my experience, if you implement sound engineering practices (thorough testing at unit, integration, and system levels, code review, clear domain models, good encapsulation of concerns, etc.), then the sort of errors that bite you are not ones that a type checker would help with. I agree that the worst Python code is probably far more unsound than the worst Go code, but I don't think that's the correct comparison; you should be comparing the Python and Go code that _you_ (or your team) would write.
I think it's easy to be dogmatic about this kind of thing; in practice most people are substituting personal preference for technical suitability. Sure, there are cases where the performance or correctness characteristics of a particular language make it more suitable than another. But for most software, then whatever your team is expert in is the best choice.
The problem was caused because I didn't know that there was a code path that passed in a bare string instead of a list, though. It's hard to write tests for situations you aren't aware of.
Because the unit tests are not to "check types", they are to check that incorrect values (e.g. a string instead of a list of strings) do not occur. They are no different from other kinds of incorrect values, like attempting to cluster an odd number of items into pairs.
> Python happily went along with it, used an uninitialized value
There is no such thing in Python. You should get NameError if a name doesn't refer to any object.
>>> def f():
... name
...
>>> f()
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<string>", line 2, in f
NameError: name 'name' is not defined
It's not my project; I'm just a collaborator. My experience has been that a very tiny minority of Python code out there is written in this style, so unless you're only starting projects from scratch, you can't benefit from it.
And that'd be fine if everyone were on board with it and that were the general direction of the project, but I don't think that's true.
I've never seen a strict, type-annotated Python project out there in the wild, and I've seen a decent amount of them. A random non-primary-contributor isn't going to have much luck stepping into an established project and getting everyone to go along with adding annotations to the whole thing.
And if I were starting a project from scratch, rather than coercing the language to do something it wasn't really designed for, I'd just use a language that has first-class support for types directly in the compiler, like Java or Go.
Agreed. I really don't understand all these buckets filth being poured on Python in this thread.
It's a first language I worked with in my life that just clicked with my brain and doesn't just drain me.
I would take a Python job over a Java/C/C++/Go/Rust any day.
There's some languages that could pull me away from Python (Nim, Crystal) but they're nowhere popular enough to move wholesale to them.
> I would take a Python job over a Java/C/C++/Go/Rust any day
it's funny, I feel the exact opposite. I work on a team that maintains a digital catalog, and a lot of what we write is about taking in asset- and metadata files, asynchronously processing them, and then publishing that to a denormalized read-optimized data store. We often joke that we mostly take data from 'over here' and put it 'over there'.
All our stuff is in Java, and honestly, if you use Lombok to squeeze out the boilerplate, and a decent dependency injection framework like Guice or Dagger, modern Java really isn't so bad. Streams are clunky but they get the job done. We use Jackson a lot to serialize/deserialize Java pojos to JSON and XML, which is pretty seamless for us so far. The Optional class is again clunky, but it works well enough.
The thing for us though is that the problems we spend most time solving are just not really related to the language we write it in. The hard problems are much more around things like operations (cd/ci, metrics, alarms, canaries), performance (latency, load, etc.) and just the nuts and bolts of the business logic (what type should this new field be? what values can it take? how do we plumb this through to downstream system X owned by team Y? etc.)
I honestly wouldn't want to have to write this stuff in Python for a simple reason: I don't think I could live without static typing, which is a fantastic tool when you need to manage a large code base written by multiple people over multiple years. I can make a change in some package, do a dry-run compile of every system that uses it, and then see what needs updating. It gives me certain guarantees about data integrity right at compile time, which is super helpful when you're doing data conversion.
But hey, different jobs, different tools. Glad you found something you're happy with.
> I honestly wouldn't want to have to write this stuff in Python for a simple reason: I don't think I could live without static typing, which is a fantastic tool when you need to manage a large code base written by multiple people over multiple years. I can make a change in some package, do a dry-run compile of every system that uses it, and then see what needs updating. It gives me certain guarantees about data integrity right at compile time, which is super helpful when you're doing data conversion.
Programming in the large without type safety is a fool’s errand.
> But hey, different jobs, different tools.
Exactly. There’s a reason your kitchen drawer isn’t full of just sporks.
> Programming in the large without type safety is a fool’s errand.
Lol. Right. No big system has ever been built in an untyped or weakly typed language. Well, except just about every bit of software we all use everyday. But it does seem like some small startups can't get by without it.
>No big system has ever been built in an untyped or weakly typed language. Well, except just about every bit of software we all use everyday. But it does seem like some small startups can't get by without it.
Many have built models of the Eiffel tower with toothpicks too, so?
You can still built things with inadequate tools: inadequate != prohibitive. You just have more problems going forward.
Which is exactly the lesson people who write large scale software have found.
What is this "just about every bit of software we all use everyday" that you wrote about as been written in weak types?
Most major software is still written in C/C++ (anything from operating systems, Photoshop, DAWs, NLEs, UNIX userland, MS and Open Office, databases, webservers, AAA games, what have you). One could use just that C/C++ software, and they'd have almost all bases covered.
The rest is e.g. Electron based software and online services. For the latter, most of the major ones (e.g. Gmail, Apple's iCloud services, Microsofts, online banks, online reservations, etc, etc) are not written in "weakly typed languages", only the client is.
And those that were initially written in a weakly typed language, e.g. Twitter with Ruby on Rails, others with Python, etc, have rewritten critical services (or entirely) to statically typed languages (e.g. Twitter went for Java/Scale, others for Go, etc).
And even for the client, most shops are now turning to Typescript (and FB to Flow) because they've found weakly typing is not good enough for large scale. So?
Python is not weakly typed. It is strongly typed in that it forbids operations that are not well-defined. For example, adding a number to a string) rather than silently attempting to make sense of them. I agree wholeheartedly about weakly typed languages, though.
I believe that marketing Python as "strongly typed" has the potential to confuse rather than educate. Python still crashes at runtime with these errors. It has nice error messages, but it still crashes, potentially in production. If you want to create your own "types", you'll have to add your own runtime checks. It's much more sane than JavaScript, but it's not strongly typed like Haskell. Python does not automatically coerce some built-in runtime values, that's it.
Not automatically coercing values is all that strong typing means. Getting a type error before you run the program is static typing. They're separate axes, and both useful to talk about in a language.
Could you elaborate or point to a resource? AFAIK, term "strongly typed" is usually used to refer to that the type cannot change but I'm failing to find a well defined definition or the comparison against statically typed.
Static typing means that types are figured out statically by looking at the source code, and type errors are detected then when it notices a mismatch. Dynamic typing means that types are worked out at runtime by looking at live objects when code operating on them executes.
Strong typing means that types cannot be substituted for other types. In C, you can write `int x = "one"` and the char * (address of) "one" is automatically converted to an int, or in Javascript you can write 1 + "2" and a string "1" is automatically created; depending who you're talking to, either or both of these qualify as weak typing.
They're both spectrums, and commonly confused with each other.
You're explaining static typing vs dynamic typing. I'm still failing to see how different Strong vs Static. If the only difference is "Static" means "types are figured out statically by looking at the source code" do you mean it's possible to change the type unlike strong typing? If not, can we say Static encapsulates Strong?
Static typing is not a superset of strong typing, they're on different axes. Strong vs weak typing (which I explained in the second paragraph) is about how strictly types need to match expected types before you get a type error. Static vs dynamic typing is about when you get a type error (during a static typechecking phase, or at runtime when you try to use a value as that type).
When you say the type cannot change, that's ambiguous: do you mean the type of the value a variable holds, or the type of the value itself? In C (a statically typed language), "int x" means that x will always hold an int, but you can still assign a pointer to it, it just turns into an int (weak typing). In Python (a dynamically typed language), the variable "x" wouldn't have a type (so it could hold an int at one point and a string later), but the value it holds does, and because it's strongly typed, it would throw a type error if you attempted to use it in a place where it wanted a different type (eg, `1 + "2"` does not turn 1 into a string or "2" into an int).
If I got this correct, you're saying strong can be compared to weak and static can be compared to dynamic. So there is no such thing as strong vs static typing comparison.
"Dynamic typing" is really just case analysis at runtime. Every static language is capable of dynamic typing, it's not some feature that statically typed languages lack. A dynamic language is really just a static language with one type.
Because most statically typed languages allow us to define our own types, add type signatures to constrain etc. Dependently typed languages also allow types to depend on values. Inference is useful, but only one aspect of static typing.
My point is that your marketing is misleading. Use "strong dynamic types" if you must, but for Python, it would be more accurate to say "strongly tagged".
C's typing is so week it might as well be an untyped language - not even a dynamically typed langue. And that's what most of the software you run every day runs on.
Static typing was all the rage 20 years ago. C++ and Java were going to save us from the chaos of C. What people found was the vast bulk of software defects are not problems that can be detected by static typing.
Static typing just created a constraining, inflexible code base, that was no more reliable than C or smalltalk or lisp. Once your beautifully conceived collection of types were demolished by the cold hard reality of changing business requirements the type system actively worked against you.
Python and ruby and javascript started gaining traction, and at first it seemed crazy to use a language that didn't have a static type checker. But after people started using them they realized they just didn't have the kinds of bugs that a static type checker would catch anyway - because those types of bugs are caught by the dynamic type checker (something C doesn't have, and C++ only sort of kind of has) at run time when you write tests. And writing tests also caught all kinds of other logic bugs that didn't have anything to do with types. They were writing software faster and more reliably in dynamically typed langues than they ever could in the old statically typed languages.
Of course no language is a silver bullet, and writing software is still hard. Combine that with the fact that our industry has no sense of history, and a fair number of programmers today have only used dynamically typed languages, and you can see why the static typing fad is coming back around.
It seems intuitive that caching these type errors at compile time rather than run time will make for a more reliable system. But history tells us otherwise. Unless you just don't run your code before pushing it to production the dynamic type checker will catch it just as well when you run tests. And your types will drift away from the reality of the business requirements grinding development to a halt.
The static typing fad has a 5 year shelf life. Just enough time for managers to force a new generation of programmers to re-write all their code in typescript or whatever and learn it is just as unreliable, and much harder to work with.
(Sound) Type systems guarantee correctness for the invariants encoded as types. If it compiles, you know it doesn't have any type related errors at all. With more evolved type systems even your program's logic (or large parts of it) is guaranteed.
Tests just allow you to test random invariants about your program. If it compiles and your add() method works when passed 2, 2 and gives 4, it still might not work for 5, 5... (contrived example: imagine it with much more complex functions, though even a simple e.g. "one line" time conversion can have similar issues).
You need to test anyway. So, is it the case that type systems provide much value beyond what a proper set of tests, which are necessary, are going to provide anyway?
If you skimp on testing your system will be crap, but at least the type system can fool you into thinking otherwise because it still compiles.
Actually, if your type system is powerful enough, you don't need to test. That's the source of the "if it compiles, 99% of the time it works right" people mention about Haskell (and even more so languages like Idris etc).
Type systems are tests -- just formal and compiler-enforced, not ad-hoc "whatever I felt like testing" tests, like unit tests are.
From there on it's up to the power of the type system. But even a simple type system like Java's makes whole classes of tests irrelevant and automatically checked.
A programmer can also leverage a simpler type system to enforce invariants in hand crafted types -- e.g. your "executeSQL" function could be made to only accept a "SafeString" type, not a "string" type, and the SafeString type could be made to only be constructed by a method that properly escapes SQL strings and params. Or the same way an Optional type ensures no null dereferences.
> Actually, if your type system is powerful enough, you don't need to test. That's the source of the "if it compiles, 99% of the time it works right" people mention about Haskell (and even more so languages like Idris etc).
Types only eliminate certain tests. You will always have system tests, acceptance tests and unit tests.
One should use types to augment their system reliability.
Haskell's type system most definitely does catch some of your logical errors. That's exactly why it is so revered.
An effective use of a type system such as Haskell's Hindley-Milner can result in a vastly smaller surface area for possible problems and thus can cut a big number of otherwise mandatory unit tests off your todo list.
>Types only eliminate certain tests. You will always have system tests, acceptance tests and unit tests.
Yes, so let's eliminate them with types, instead of doing them. "Acceptance tests" are not concerned with programming.
>Types will not catch logical errors in your code.
Actually, depending on the type system, it will.
That's how program logic is tested as "proof" and programs, implementations of algorithms are determined to be logically correct in more exotic languages (but even in C + some restrictions + the right statically checking tooling, NASA/JPL style project do that).
The question is not whether a type system will catch bugs. The question is whether a type system finds enough bugs that tests (sufficient to cover the things that the type system does not catch) would not also catch.
If you have to point to something like Idris I don't think you're making a real world argument yet.
Both static type systems and unit testing are just tools which are supposed to help programmers to deliver higher quality software.
Both static type systems and unit testing have their disadvantages. For static type systems, you sometimes need to bend backward to make it accept your code and it's not very useful before the code grows large enough. For unit tests, even if you have 100% test coverage, it doesn't mean that you're safe - underlying libraries may behave in unexpected ways and the test data input won't ever cover the whole range of values that the code expects to work. Integration tests have the same problem, the prepared input represents just a few cases, plus they are generally harder to run so they are run less frequently.
So, both tools are useful but they aren't solutions for all the problems in programming. Static type systems have the advantage of being checked without running any code, which should be much quicker than running the tests. Static type systems become more useful as you increase the precision of types and the amount of annotated code in the project. When used correctly, they provide certain guarantees about the code which you can rely on and they are used to restrict the domain (set of possible inputs) of type-checked procedures and classes. This means that you can write fewer unit tests because you don't have to worry about certain conditions which the type system guards against (static guarantee of something never being `null` is quite nice).
Anyway, I think that both static type systems and tests are great tools and they can and should be used together if you value the quality of the code you write. This is getting easier thanks to gradual type systems (optional type annotations like in Python or JS) which allow you to get some of the static guarantees without insisting on everything around being typed. With tests and mypy (in Python) you're much better off in terms of code quality than if you used just one of them. I see no reason not to use them both.
> For static type systems, you sometimes need to bend backward
> to make it accept your code and it's not very useful before
> the code grows large enough.
How large need a program to become, before the advantage of being allowed to write fishy code is counter-balanced by the types becoming untractable and the code impossible to refactor in any meaningful way?
This is a serious question. Some years ago, apparently Guido Van Rossum though 200 lines would be already quite an achievement [0].
Based on my own experience, I feel that 99 out of 100 errors thrown at me at compile time are valid and would have caused a crash at runtime (ie. when I do not expect it and have lost all the context of the code change). And I get about 50 such compilation errors in a day of work, so I guess I could write without the compiler safety net for about 10 minutes. That's my limit.
One could object that a 10 minutes program written in python can accomplish much more than a 10 minutes program written in Java. That's much certain! But then we are no longer comparing the merits of compile time vs runtime type checking, but two completely different languages. Of course it is easier to write a powerful/abstract language with runtime type checks, while writing a compiler for a powerful language is much harder. Still, since (and even before) python/perl/php were invented many powerful compiled languages have appeared thanks to PL research, that are almost as expressive as script languages. So it would be unfair to equate runtime type checking with lack of expressive power.
Now of course tests are important too. Compile time type checking does not contradict testing, like you made it sound somewhat in your message. Actually, if anything, it helps to test (because of test case generators based on type knowledge to exercice corner cases).
I'm sorry if all this sounds condescending. I am yet to decide whether I should allow myself to sound condescending as the only benefit of age :)
But I'd not want to sound like I'm upset against anyone. Actually, I'm happy people have been using script languages since the 90s, for the same reason I have been happy that many smart people used Windows: my taste for independence gave me by chance a head start that I'm afraid would have been much tougher to get based on my intelligence alone.
And now that static type checking is fashionable again I'm both relieved and worried.
> Some years ago, apparently Guido Van Rossum though 200 lines
I think it's better to measure the number of separate code entities (classes and functions and modules in Python) and how many different use-cases (ways of calling functions and object constructors) each entity is expected to cover... After converting to LOC, I'd say ~500 would be the limit. After that, it's a constant fight with TypeErrors, NameErrors, and AttributeErrors - it's just that everyone is already used to this, while not many know of any alternatives. Also, there are substantial differences between languages - in some 10 lines are enough to start complaining, while in some others I've seen and worked with ~2k loc code and it was manageable.
> many powerful compiled languages have appeared thanks to PL research, that are almost as expressive as script languages.
Yes, but on the other hand, some powerful static type systems for dynamic languages also appeared, and some of them are close to Haskell in terms of expressivity. The particular example here would be Typed Racket, which has a state of the art type system which is built on top of untyped Racket. It supports incrementally moving your untyped code to the typed one (whether a module is statically or dynamically typed is decided when module is created; as you can define many (sub)modules in a single file, you can just create a typed submodule, re-export everything that's inside, and move your code there one procedure at a time). Also, it automatically adds contracts based on static types, so that they still provide some guarantees when a typed function is imported and used in untyped code. There are many interesting papers on this, and TypedRacked is really worth looking into, if you have nothing against Lisps.
> Compile time type checking does not contradict testing, like you made it sound somewhat in your message.
Damn! I actually wanted to argue exactly this: that both tools are useful and both can be used together to cover their respective weaknesses. :) Looks like I need to work harder on my writing skills...
> I'm sorry if all this sounds condescending. I am yet to decide whether I should allow myself to sound condescending as the only benefit of age :)
Well, it didn't sound condescending to me, so no prob :) But, if you'd like an advice on this: please don't try to be condescending on the basis of age alone! It's totally ok to sound condescending if you have knowledge, experience and skill to back it up... Well, at least in my book :)
> Programming in the large without tests is a fool's errand. Type systems don't guarantee correctness.
I never said you do not need tests nor that static typing is a panacea. In my view it's a necessary, but not sufficient condition, when programming in the large.
No but they help. You can find figures of a 15%-38% reduction in bugs for TypeScript versus JavaScript. So that does not consider the additional effect of strong versus weak typing.
It's a bit off-topic, but I wanted to comment on this:
> a simple reason: I don't think I could live without static typing
The gradual type system for Python (mypy at the moment) is actually a very good tool. It's as expressive as C# easily, despite some limitations. It fully supports generics and protocols (interfaces or traits in other languages), it allows the user to control the variance of generic arguments, it supports pretty accurate type inference (although not as powerful as OCaml), and so on. Just set up a CI where one of the build steps is running mypy and make the build crash if there's an untyped and not type-inferrable statement anywhere. This is what I've been doing for a year already and it really helps with the maintenance of the projects and with development once the codebase becomes large enough.
This may be as good a chance as any to say this: gradual type systems are here to stay. It's been more than 10 years since the original paper (J. Siek paper was written in 2006; the PLT Scheme (Racket now) guys started working on what became Typed Racket around that time too) - as usual, the industry lags behind the research significantly, but it's bound to catch up at some point. Facebook's Flow and Mypy are the first large scale industrial applications (if I remember correctly) of the theory, but I believe it won't be long before similar functionality pops up all over the place.
While there's still much to be done (like deriving run-time constructs from static type annotations and preserving at least some of the benefits of static typing when interacting with untyped code), these type systems are already powerful tools, and the fact that they are "optional" isn't really a problem for bigger projects, where it can be enforced by the build process. Currently, the lack of type annotations in external libraries poses a problem, but the number of annotated ones is bound to grow because the static type system is an incredibly helpful tool if used correctly and consequently.
So, what I want to say is the distinction between statically and dynamically typed languages will continue to blur and, at some point, will become irrelevant. Especially when you notice that many statically typed langs started to also grow features from the "other side" like marking a variable `dynamic` and allowing the user to do whatever they want with it without complaining.
We have the typing enforced as mandatory for all new code in our codebase (and have progressively been retrofitting it tonokd code as we touch it). It's saved our asses many, many times.
Interesting that you mention the difficult problems being around CI/CD and operations. I had to get our Python application’s CI/CD pipeline off the ground and it was much harder than it would have been in Go, for example. Notably, figuring out how to manage dependencies and run tests in a way that was reasonably performant was a massive challenge. We made the mistake of using pipenv, but downloading dependencies took half an hour. We should use something like Bazel to solve those problems, but it doesn’t support Python3 (allegedly some folks have hacked things together to get it working, but I haven’t managed to reproduce it). Further, packing dependencies into a lambda function is tough because Python libs are often bloated and static analysis tools are lacking, making it hard to trim the tree. I’m sure they’re are solutions, but they’re hard to find relative to Go. Not sure about Java or other languages.
>All our stuff is in Java, and honestly, if you use Lombok to squeeze out the boilerplate, and a decent dependency injection framework like Guice or Dagger, modern Java really isn't so bad.
So, basically, if you go out of your way not to use Java as is, Java is not so bad for the task?
I've worked in so many languages and environments in my career and Django/python/virtualenv has to be one of the least painful. I tried Rails which is very similar but feels "inside out", a good friend of mine loves Rails and hates Django and has the exact same feeling about Django.
That's kind of my point, you may like other environments better, such as React/Node/NPM but that doesn't mean Python is a horror show.
Yes, there are very, very large working python codebases in fields out there that demand correctness. I'm honestly getting tired of the static typing circlejerk that has entered the industry.
Btw, jpmorgan dev team is rediculous because support is so massive the only way new projects get done is by hiring massive amounts of people / consultant firms and then lay offs.
Not saying there is a problem, I’m sure some people like to have their throat taken out when the trade doesn’t execute at markdown price.
Anyways I don’t have a problem with that type of pressured enviorment. I’m moreso pointing out people’s need for comfort of solution rather then sustainability. Getting started is more difficult so many are turned away.
Also some of the computation python can do is very powerful and I would trust it if I was no risk besides myself going balls deep.
Python is the first language that clicked with my brain as well and in college I often used it to prototype homework algorithms before translating them into the language I needed to actually submit my work in. I have nothing but love for python as a language.
At the same time even when I used it heavily I never saw it as anything more than a scripting language to sit in front of some tool that was written in a language I couldn't be fucked to learn at that moment (numpy and scipy were used heavily throughout my college career).
If I'm being honest I don't understand how anyone could get as worked up about a language as the people in this thread have. At the end of the day most of us are still writing unportable imperative code that runs like shit. Maybe blaming language is how we cope with our own failure as engineers.
> At the end of the day most of us are still writing unportable imperative code that runs like shit. Maybe blaming language is how we cope with our own failure as engineers.
Python is great; I picked it up back in the early 2.xx days. My main problem with it is the string handling/conversion code is brittle and the breakage of backward compatibility. But it's overall a great language.
And some people feel the opposite. I’m glad Python works for you. I had the same click-with-my-brain feeling with Ruby, whereas I find working with Python to be draining and demoralizing.
> I would take a Python job over a Java/C/C++/Go/Rust any day.
Why do you group those languages like they’re similar but different from nim and crystal? They’re wildly different in terms of their target domain, runtime models, etc. Go and Java are general purpose application languages and the others are more suited for systems or performance critical applications.
I had to learn Fortran IV for my first job. Am I allowed to hate Python?
Are you assuming everyone complaining here is young, and this is their first language? Consider that maybe they're complaining because they've used older languages they liked more.
Often, not having a feature is preferable to having a feature designed or implemented poorly.
Sure! I dislike all sorts of languages and environments.
That wasn't my point. My point is if you had to write Fortran IV using an IBM 32xx terminal you wouldn't be quite so hyperbolic about modern Python.
Unless you are claiming you would rather return to writing Fortran IV than use Python because you like Fortran IV better, in which case I'm very confused.
Just because languages were more of a PITA in the past, doesn’t mean we shouldn’t pick out faults of current languages and search for new/better solutions...
Well, the original context here was about comparing python to fortran, which also doesn't fit this "competitor" criteria. It's an apples to oranges comparison, sure, but that's the way this whole discussion started. At least Lisp is roughly the same age as fortran, which gets at the root assumption that python is an improvement over older languages.
There's a difference between hating Python, and saying (I'm guessing is the comment that spurred this one) this: "I try to be a good sport about it, but every time I write python I want to quit software engineering.", like a top-level comment below says.
If you had to write Python, would you also want to quit software engineering? Would you go back to Fortran instead of Python?
Of course you're allowed to hate Python but someone saying "every time I write python I want to quit software" is either extreme hyperbole, some tangentially related issue like depression, or just no language at all would make them happy enough.
In my opinion, as a computational researcher, Python was not really meant to be a Scientific Computing Programming environment. It was a big historical mistake to go in that direction. In the near future, hopefully, it will be replaced, by a better alternative. and believe me, for most people who do not speak highly of Fortran, when it comes to developing a new language for scientific computing, they pretty-much end up reinventing Fortran.
Python 1.4 was an awesomely simple programming environment and I pretty much immediately fell in love with it. Then features were added. Now it is a whole home improvement store full of kitchen sinks.
I think that programming is a sort of theological process. Popular languages attract ideas. Unfortunately, in the case of Python, those ideas were not effectively filtered and now we have an expression of as many ideas as can possibly fit. The ultimate design by committee...
I suspect that the recent excitement about assignment expressions is really a kind of straw that broke the camels back. The problem isn't just this one feature, it's the sum of them.
>I think that programming is a sort of theological process. Popular languages attract ideas. Unfortunately, in the case of Python, those ideas were not effectively filtered and now we have an expression of as many ideas as can possibly fit. The ultimate design by committee...
It's funny, a lot of people hate on Elm for the exact opposite reasons: one person dictating the language's direction and removing features. I suppose a nice balance could be struck between the two ends of the spectrum.
>Python 1.4 was an awesomely simple programming environment and I pretty much immediately fell in love with it. Then features were added. Now it is a whole home improvement store full of kitchen sinks.
I've used Python at the time (and up to now). It was a revelation compared to Perl, but it sucked compared to modern Python.
I think it's pretty disappointing that most of the top comments don't talk about the interview with Guido himself over the history of Python. Tangentially related discussion is one of the appeals of HN but I think it's a bit out of control here.
Well in just glad this is the top comment, as Python really is taking over the world for a reason.
And of all the bugs I have written in recent memory, not one came down to a lack of static typing. They were due simply to logic errors, flawed assumptions, misunderstood requirements, and good old race conditions. The static typing zealots like to think if it compiles is must be perfect, however this is a mirage. Unit tests in Python can compensate quite well for lack of static typing.
Have you ever worked in a large engineering organization full of engineers with varying degrees of experience all trying to accomplish the same goal?
I can't imagine anyone has ever tried to do engineering at scale (people wise) and did not find the value in static typing.
It's why startups eventually moved off RoR once they started scaling. It's why there is such a large push to type JavaScript (have you seen the rollbar article about the top 10 errors in JavaScript? All but one have to do with types: https://rollbar.com/blog/top-10-javascript-errors/), it's why Facebook created Hack, and outside of parentheses repulsion, it's probably why so few large projects have been written in a LISP or LISP descendant.
Python is great for small: small teams, small organizations, small projects with a few dedicated tasks, small scripting tasks. Most people aren't trying to take anything away from python here in the comments save a few irrational responses.
*again want to stress in my comment when I speak of scale I mean scaling people wise: more organizational structures in your company, more engineers, more collaboration between teams.
>I can't imagine anyone has ever tried to do engineering at scale (people wise) and did not find the value in static typing.
>why so few large projects have been written in a LISP or LISP descendant
The major dialect of Lisp, Common Lisp, is strongly typed, and many large projects have been written in it, for CAD/CAM, controlling a NASA spaceship, complete operating systems (Open Genera), the Mirai 3D graphics suite used for creating Gollum in "the lord of the rings", etc.
> 1. Uncaught TypeError: Cannot read property
If you’re a JavaScript developer, you’ve probably seen this error more than you care to admit. This one occurs in Chrome when you read a property or call a method on an undefined object.
Does typing stop null object errors in JS, Java or C for that matter? No. You need to continually check for null objects in all langs I use including Python. It seems most of the bugs on that page are of a similar vein.
Null reference errors in Java and C are due to The Billion Dollar Mistake, which is a specific deliberate weakening of a static type system. Statically-typed languages that do not commit The Billion Dollar Mistake do not have null reference errors.
Just wanted to mention that you can selectively statically type variables with the cython library. Of course using cython also changes other things and requires compilation, but i have found that it generally just works.
I cut my teeth in FORTRAN IV (on RSX-11M). I lived through the archie days of uuencoded fragments to build my C environment, supplemented by DECUS tapes. Those were good old days.
I use Python 3 these days for a lot of stuff. It's pretty good.
These are better days. We all have complaints, but on the whole, things are not too bad.
I think for the most part that when nothing meets your expectations, it may be that your expectations need to be adjusted.
People don't realize what a revelation Python 1.x was back in the day. Around 97 or so I was tasked with porting a giant mathematica program for calculation diffraction grating efficiencies into something which ran open source (there was no free mathematica engine for running scripts pack then). Back then, that meant either C or Fortran. Sure, stuff like Perl existed; nobody thought of it as a real interpreter that could be used to construct complicated things any more than Awk was. When I realized Hugunin over at Livermore had done lapack extensions for Python (whatever numpy was called back then) ... well this massive job was done in a week and worked the first time.
The winning thing python had that nothing else had at the time was it was social, it was readable, and there was generally only one way to do things: the right way. It no longer has the latter quality, and the preferred coding style in it seems to be java-ish OO-spaghetti, but it's still pretty good.
That said, these days, I resent every damn time I have to use it. It's eating data science, more or less because pandas and scikit is ... mostly good enough, and because unlike R it's .... mostly good enough to deploy in an enterprise application. But if you're working on the exploratory side of data science, Python is shit compared to R. Doesn't have the tooling, doesn't have native facilities, and is vastly more long winded. All the attempts to make Python more ... X ... are probably a mistake also. You're taking a beautifully simple tool and making it more exotic and complex. It's like trying to use Matlab to build webservers.
> ... Perl existed; nobody thought of it as a real interpreter that could be used to construct complicated things any more than Awk was ...
I can't help but to gently interject here. By 1997, I'd been programming for some time, and there certainly were people who consider Perl suitable for programming in the large, and there were certainly big projects so written.
While I disagree with the specific word 'nobody', I agree with the sentiment: Perl was widely considered to be only useful for 'small' things at the time. Widely, but not exclusively.
This is spot on. In my current workplace we use Clojure, and Clojure has many of the same problems in package management as Python does (no lockfile, no easy way to create reproducible builds, no way to declare range of dependencies unless you use version pin, etc. etc.).
However, I never saw any complaints about Clojure package management in any topic about Clojure here.
We use Clojure massively enough to have multiple issues with dependencies, including the fact that sometimes we need to build a new version of a library simple to build it with more recent versions of dependency X, for example.
It is not that bad, much like I also don't think Python packaging bad. Other ecosystems have better solutions, though.
I'm a somewhat older programmer, and I've worked with a variety of languages (C, OCaml, C++, Scheme, Go, Java...). I think all of them are great in their own way and there's a lot to be learned with all of them.
I started to use Python quite recently and I really like it. It is a well-designed language with high-level abstractions that are really fun to use. I like the pervasive use of iterators, the 'everything is an object' philosophy, the minimalist syntax, the build-in datatypes...
That being said, I feel that the dynamic types show their limits when projects getter big. I use linters and static type annotations but I find refactoring very error-prone and there's a point where I don't really trust my programs.
I always thought that sentiment to be too broadly applied. A good craftsman should be able to make use of the tools he is given, but nonetheless not shirk his duty to improve upon them.
Not to say I have any real complaints about Python.
I don't think Python is a bad language but that quote is a pretty ridiculous response to criticism when the discussion on this article is a far cry from people blaming failures on Python/Python tooling.
No, not at all, I'm saying if you had been a C programmer in the 90's you would have some perspective on some of the complaints and comments about python in 2019.
Why are you complaining about C in the 90s? If you'd been using punchcards in the 50s you'd have some perspective on some of the complaints and comments about C in 1990.
I'm actually not complaining about C in the '90s, it was amazing compared to Fortran in the '80s.
Funny story, my first co-op job (Fortran IV), my boss made me fix a bug using punch cards so I'd appreciate why the codebase wasn't as nice as it might be. THAT was prespective.
Fortran IV? But yeah, Fortran was the goto tool for mathematics, simulations, etc. for the same reason Python often is today: libraries and existing code. I had to convert a simulated annealing algorithm from Fortran to C in the '90s
Why would you rationally compare something that exists and lives today, in the form it has today - with something from early 90s? In what world is that an objective comparison?
Back in my day we had to walk 20 miles to get to school -- uphill both ways!
As if the only way to gauge quality is to forever compare to the technologies of the past. Python was a great improvement on its peers in its heyday, but people, technologies and philosophies have changed since its inception and there is nothing wrong with wanting something more. That's how progress is made.
I find programming in Python dull and a little mind numbing. Sure, maybe Python was an okay choice 20 years ago, but the world has left it behind at this point. It’s lack of functional programming constructs, very few data structures in the stdlib, terrible performance, worthless statement vs expression semantics, no multi threading, no macros, etc.
Instead of evolving into something okay, Python is pretty much the same broken language as it’s always been, run by a guy who is openly hostile to PL theory, FP, and any major changes to Python whatsoever.
If you ever need/want to use Python, do yourself a favor and use Racket instead. Racket is better than Python in every single way (unless you are doing data science with Python, in which case fine keep using it).
Isn't it also just as much about Python is having it's day, granted a day long in the waiting but many langs go through this (Ruby, PHP) and then it tapers off and the next language has it's day.
> Probably Go will be the next hotness in 5 years.
I think it will be difficult to grow a large ecosystem for a language with very poor FFI performance [0] in the long run. Golang's poor FFI performance is the number 1 reason I wouldn't use it for my own projects.
I think it's less an issue of the average programmer using FFI and more an issue of common libraries leveraging it.
With python, why is it so popular currently? For large part because of its very good data science and machine learning ecosystem. And why does that exist? Mostly because python libraries like numpy, theano, and scikit-learn were built on top of mature, high-performance C libraries like OpenBLAS, LAPACK, and Cuda.
I very much doubt that anything like scipy would exist if the developers had to reinvent the wheel of the underlying numbers libraries from scratch. C's been around a long-time. There's huge amounts of high-quality mature software that already exists in a C framework. A language's ability to easily "plug-in" to the C ecosystem is a major leg-up when it comes to bootstrapping its own comprehensive library ecosystem.
I've run a few semi popular open source projects, and it's surprisingly common to hear people tell me it's useless because it's lacking their specific pet feature. Now these comments just make me laugh, just like the parent's gripe with slow FFI
I think you don't get the point. Most people don't create such interfaces, but the libraries/systems they work with use them a lot, at least in other languages.
This is for example the reason why there won't be a large mathy/scientific ecosystem in golang.
Python needs FFI, because its native performance is fairly poor [1]. In general, Go gets away with less FFI because it's fast enough that it doesn't need things to be implemented in C to run quickly. This is especially true if you consider "Go" as "Go + its ASM", which is probably what you'd want if you think of it in terms of science programming.
For another example of a similar effect, Rust has great FFI. Yet I would expect over time it will be necessary for fewer and fewer things, because Rust is already roughly on par with C, and over time, a native Rust API will still be preferable to a C API wrapped with a Rust access layer. It will always have great C FFI, by its nature, but the percentage of projects that won't need it is already pretty high and probably only going up over time.
[1]: People seem to misinterpret this statement a lot, as if I'm saying Python is bad or something. No; it is simply this: Python performance is not very good at a very raw level. It is merely one characteristic of a language out of many, many relevant ones, not a full assessment of the language. Python has many other dimensions in which it has superior capabilities. It just pays for that on the performance dimension. (Whether that's an essential or an accidental tradeoff, well, ask me again in ten years; the programming language community seems to be in the process of working that out right now.)
It's not just about speed. Great mature libraries have been written in C over the decades. Rewriting them in a new language is a herculean effort. Writing bindings to them lets you use the fruit of all that labor.
What you say is true, yet, I observe that people undertake that herculean effort with some frequency.
Whether that's a good idea, and why that is, those would be entirely separate conversations. But it's just an observable fact that languages pick up native implementations of core functionality over time, subject to certain performance restrictions (e.g., I'm sure that if it wouldn't be unusably slow, Python would have a native-Python image library... it's just Python doesn't really have that option).
The effort I'm familiar with is the many attempts to make a decent linear algebra library in haskell. There are a number of libraries with huge work put in, but none has reached blas/lapack parity. Nothing's remotely comparable to numpy in ease of use yet. It picks up more libraries as time goes on, and the existing ones mature, but it's so slow that I'm skeptical they'll ever match numpy's usefulness a decade ago.
I think it's really easy to focus on the big hitters and forget they are the exceptions. Yes, matching lapack, any GUI toolkit, a browser engine, and a handful of other things is a big challenge that takes its own community to overcome, not something any language community can do with an incidental fraction of the community's available firepower.
But those are the exceptions, and often you don't need a best-of-breed solution and may prefer the language-native one.
Again, I'm not theorizing about what could be here; I'm looking out in the world, where I see that most libraries tend to emerge out into a native version if the underlying language can possibly meet the basic requirements for performance and such. This is something that needs to be explained, not explained away.
Also... I love me some Haskell, and on a per capita basis the community is great, but if Rust's community isn't already several times larger and growing faster, I'd be stunned, just to pick one example. Haskell has some very interesting cases of best-of-breed libraries, but it doesn't exhibit the library profusion you get from sheer personpower. (Of course, it doesn't really have the problems you get when your libraries are generated by sheer personpower either.)
If the go-silo offers everything you need than yes, there is no need for a good FFI in go... but if you want access to the low-level ecosystems of C/C++/Rust without a heavy penalty, then it might not be an option.
Actually, Golang isn't so great. Try to change something lower level, for example in their socket implementation. Also, it's trying to promise a sane concurrency and all code I've seen use mutex all over the place.
> Also, it's trying to promise a sane concurrency and all code I've seen use mutex all over the place.
I think that's a fault of the devs, more than the language. In many cases (not all, of course) Go gives you multiple ways to achieve concurrent safety. Channels, being the big secondary. Yet it generally (in my experience) requires a very different implementation and in general has a lot of pitfalls. Over all I don't like Go these days, but I prefer it over Python (mostly due to at least having basic types)
I've switched everything to Rust though. Just as productive as Go (to me), with more tooling. Though, Rust will be much better in a couple years with some additional baking on new features (GATs, Futures, etc).
> I don't like Go these days, but I prefer it over Python (mostly due to at least having basic types)
You should try type annotations, with mypy and/or PyCharm it helps finding bugs before you run the code. It also makes autocomplete and refactoring work correctly (I would say they work better in PyCharm than in GoLand)
As someone with a tendency towards writing my own custom solutions (from scratch or forked), I'm learning the hard way why people prefer depending on premade packages: they're usually better documented, better tested (both written tests and real-life battle-tested), and in a team/collaborative environment, having some canonical packages that everyone is familiar with helps to have common understanding, rather than having to explain how a one-off custom solution works.
1. the package offering the functionality I needed was much more complex than needed and my solution implementing that functionality was much simpler (because it implemented only what I needed)
2. was not the best quality, because it was written by someone who was not necessarily better than me, or didn't spent enough time understanding the problem he was trying to solve
Having said that generally popular packages are good quality, although even then #1 applies, if you need just small functionality of a specific package try implement it yourself, it might turn out that the problem was not as hard as you thought and because you're just implementing what you need it might be more elegant.
Yes and no. Go has its applications, but is not built to replace python but rather to be a better c. That overlaps with the "easy code" part and not much else. Glue code, application scripting, etc. are python's strengths, and I don't see those going away.
Source on your claim? The best I was able to find is this:
> Go attempts to combine the development speed of working in a dynamic language like Python with the performance and safety of a compiled language like C or C++.
Maybe, I love Go but I'm not sure it's the next "hot" thing, all the recent "hot" languages have been scripting languages PHP, Ruby, Python, JS ... I'd say JS and Python are currently jockeying for that position.
> all the recent "hot" languages have been scripting languages
All the languages you list are over 20 years old. I think the popularity of Go and Rust show that statically-typed languages are having a resurgence.
As a long-time C programmer (with significant bits of bash and perl on the side), I've really enjoyed learning Go. But the responsiveness to developing new language features has been _sooo sloooow_. We'll have to see in 10 years if this turns out to have been a wise decision or not.
Isn’t JS one of the most popular languages? It has a monopoly in the browser so that and C++ already has the most impact on users. Hard to see how it can become more popular.
I'm curious, do you mean compiling to JS and running it on the servers? Because I would love for the runtime itself to run TypeScript, something like Deno [0] but (maybe in the future) as mature as Node.js.
I was a PHP hater before but in my last job, I had to rewrite a big chunk of code from PHP 5.x to 7.2. Actually quite liked it. I would happily work with PHP 7.2+ code in any day but still wouldn't start a new project in it.
Yeh I recall working on MAP/Reduce using Fortran/ PL1G (1980's) we had to build an entire suite of JCL programs to mange every thing including build / deploy.
I don't agree at all. Your comment basically says: "It used to suck badly. So don't complain it sucks now.". I think dynamic typing is the bane of good software and we as an industry should try to actively discourage new code to be written in dynamically typed languages.
That's not to say that Python doesn't have it's place. But I see it more as a programming language for small utilities no more then 2k loc in length.
No, what I'm saying is that it used to suck really badly and we survived just fine, so I find all the over-wrought hand wringing about the havoc and burnout caused by Python's flaws hyperbolic.
No we damn well DID NOT. This is one of the most infuriating lines used all over for dismissing concerns about modern things and developments. "Oh why do we need vaccines/antibiotics, we survived just fine without them" except, you know, for all the hundreds of millions of deaths. Minor detail that. Mere "survival" through gross inefficiency is not a real yardstick.
By the same token, "it used to suck really badly" in programming and the result was awful practices, crashing, and security problems out the wazoo. We sort of "survived" that by a mixture of just plain eating the heavy losses and having them be somewhat mitigated by virtue of simply having less surface area for damage since less stuff was tech based or connected. Infrastructure was less built up. But times have changed and standards for, and value of, security/stability/supportability have increased dramatically.
Yes in a sufficiently large group I'm sure you'll be able to find individuals engaging in hyperbole about any such thing on the internet. But that doesn't in turn mean that there aren't very real, very serious concerns raised around the context of modern practice. Dismissiveness based on bad historic practice is not merely uncalled for, it's just plain weird.
Edit to add: another issue a lot of this dismissive comments tend to ignore is cost & skill. Yes, great things have been done with options we'd now consider subpar in the past, that's what they had to work with. But those things were done by the best of the best, with huge budgets, lots of experience and so on. A very important part of advancement is allowing the "same thing" to be done more cheaply by more people, and in turn be used for wider array of applications.
All the languages you mentioned are statically typed though so they already have a large advantage compared to Python. To be honest if I can choose to inherit a 1M LoC code base I would even choose COBOL over Python.
To understand the hate, you have to realise that no one likes being forced into using a particular technology. Especially one that is more of a lowest common denominator and ignores much of the progress in programming language research over the last 40-50 years (e.g. expressive static type systems, and Python still markets itself as "strongly typed").
> and Python still markets itself as "strongly typed"
What's wrong with that? It is strongly typed (doesn't coerce everything like Javascript) just not statically typed (objects have types but variables don't).
I go with what helps me get work done. I have a feeling many people are the same. Python lets me be productive in a crazy variety of tasks and mostly gets out of my way when I do so.
In my industry, the companies that have standardised on Python and consequently now have large Python codebases are not very productive environments anymore. Perhaps they were once, for the first few people, in the first few months.
It's no surprise that the Python community has now started to try and retrofit types.
Why do you think that Python should be compared to Fortran? Why not Rust or Julia? If we compared everything to worse humanity would not progress at all. I have spent 10 years on Python and I sympathize with some of the criticism below here pretty much. I wish Rust or Julia will replace it for data crunching soon.
I love python as a scratch pad for playing around with code, but I don’t think I would put anything into production written in it. It’s just too hard to debug and maintain and deploy once it gets to even a medium amount of complexity.
Interactive python and Jupiter notebooks are an absolute joy to work with though.
A hundred thousand years from now, when the Terran Empire's Dyson Swarms are ubiquitous throughout the Orion Arm, the relativistic generation-ships of the Andromeda Colonization Fleet have set out on their multi-million-year journey across the intergalactic deeps, and the World Computers housing the Great Intelligences serve the daily needs of quadrillions of citizens, there will still be job ads for COBOL programmers.
Python's PIP? Or do you mean Anaconda? Or VirtualEnv? Or Poetry, which someone here has pointed to?
Also - you're moving the rhetorical goalpost. First you claimed there was no package management system, no you're complaining about the lack of default.
Technically he's wrong, and you are right, in that there are package management systems for C++. Practically, your being technically correct does not matter.
No package management for C++ is popular enough for C++ programmers to avoid all the hardworks of manually managing packages, because there are always some (or most) packages you rely on not in these package management systems.
pip, for all its shortcomings, does include almost all of the Python packages you ever need to use. That is a significant different in practice.
Given that OP was talking about getting C/C++ packages "off a usenet archive", I think it's clear that they were talking about what coding C and C++ was like two or more decades ago.
What about pkgconf + autoconf? Might not be the grand unified package manager you're conditioned to look for, but works well and is pretty much universally used. Besides, C and C++ aren't about language ecosystem lock-in, but about creating standardized artifacts (shared libs, static libs, and executables) designed to work and link well in a polyglot environment (well C++ maybe less so with its symbol mangling/typesafe linkage).
I don't think its hyperbolic. If so many people are complaining then we have a issue. I have tried to learn Python but every time I did, some things kept turning me off.
- First indentation was a issue for me but I looked past it and went ahead to give another go at Python.
- Even the best in class IDE suffer to give any kind of insight into python code.
- Two versions: At my workplace we use Python 2. I prefer to learn newer version but don't have choice. When Python 3 came out they should have initiated deprecation of Python 2 but that does not seem to be the case. Tonnes of libraries are still in Python 2. They should have maintain language backward compatible instead of fragmenting the entire community.
- So many ways to do a thing. This is subjective but I need a language which gives predictable performance for given piece of code. Go does this. There's usually one way of doing things and only one way. No need to know nooks and crannies of the language nor there is such a thing in Go
You’re probably aware of this, but at the odd chance you aren’t, Python 2 end of life is 2020. You really should be moving to Python 3.
> Go does this.
There is a lot of love for Go and Rust on HN, but unless your region of the world is significantly different than mine, then chances are that there won’t be a Go or Rust job in your lifetime. I’ve only ever seen one of them mentioned in a job opening for my entire country, and that was as “nice to know” for a c++ job at Google.
I’m sure Rust and Go are truly excellent languages, but that doesn’t really matter for most people, if they never manage to see any adoption outside of Silicon Valley.
So true. There are tonne of Python, Java and other legacy software jobs compared to Go.
I am probably gonna have to learn Python (which I think is better than existing legacy technologies) because of this though I am not really interested.
On January 1, 2020 will happen ABSOLUTELY NOTHING.
It will be gradual, as long as you're developing more and more packages will refuse to work. If you realize there's a bug in one of your dependency and the fix is in a version that doesn't work on python 2, tough luck you either will have to backport the fix or migrate the code.
As time progress it will be more and more work to deal.
And guess what, some packages didn't even want to wait they already dropped python 2 support: https://python3statement.org/ (look at Projects Timeline) so it is starting happening right now. I'm wondering if pip will decide to drop the support in 2020, that might end up being the biggest hit.
Complaining isn't the problem, yes there are package management problems, a virtualenv package solves most of them. I'm not saying there isn't. Python 2 really stopped being a problem for me last year, I haven't hit an issue for a while.
I'm talking about the comments saying it was causing their burnout "x100" and other such hyperbolic statements.
According to people who work with a lot of programmes teaching coding, the reason python is so appealing to new coders is the syntax. Particularly the lack of braces, the indents, the nice keywords, that make reading code much easier to a newcomer.
Having taught both python and JavaScript, I can tell you that the former is far far less confusing to newcomers than the latter. Imagine explaining the problem with equality comparison or prototypes in JS to someone who just learnt what an IF statement is.
The reason I agree that python will dominate programming for decades to come is the batteries included and pythonic approach that enables people to do cool things with just a few lines of code. As a result of the above, the ecosystem has reached critical mass. For most tasks there’s a library that does that, and it’s good, and it’s easy to use, and the developer of the library usually doesn’t expect you to understand their opinionated philosophy to use it.
I love the python convention of putting a minimal example on the homepage of a library.
You’re not going to get your average developer to appreciate the beauty of Lisp or the value of static typing. They want their ML to work, or their graphics to display, that’s all. Ends not means, almost entirely, for most people who aren’t full-time programmers.
Let’s not hate on a language that opens up programming to the world, unless we want to be gatekeepers. If you want to maintain an air of superiority, just learn Lisp or Haskell or C++ and write code that’s “deeper” into the tech labyrinth. Programming will probably specialise into “end user programming” and “Service and system programmers”. Embrace the democratisation of coding.
Since most of my professional experience is with C like syntax languages, I don't get how the lack of braces is an advantages, quite the opposite in my limited experience with Python.
Braces define scope unequivocally, they are easy to visually parse and don't care whether you are using tabs or spaces or even if you, loud gasp, mix them. Furthermore, you can copy and paste, say for loop from a method to another method and it will work. In Python you might have to faff about with spaces (I'm sure there must be IDEs that solve this problem but it was no fun on vim or even notepad++)
Absolutely agree. maybe its just cos I am coming from languages with braces (java) but yeah I find the lack of braces + whitespace having meaning (tabs versus spaces for example) _incredibly_ frustrating
I also don't like the fact it is dynamically typed. It makes reading soure code so much harder. E.g What is the type of this thing that is being passed as a function arg?
> I also don't like the fact it is dynamically typed. It makes reading soure code so much harder. E.g What is the type of this thing that is being passed as a function arg?
I didn't want to get into that as it always seems to open a massive can of worms but agree 100% with you.
I need all the help I can get when developing, to throw away all the type checking that the compiler does before runtime seems like developing with a hand tied behind one's back, but each to their own, far be it from me to try to convince somebody that they should switch languages: if Python works for you, then great, keep using it.
So does whitespace, otherwise Python wouldn’t work as a programming language
> they are easy to visually parse
only if you indent properly
> Furthermore, you can copy and paste, say for loop from a method to another method and it will work. In Python you might have to faff about with spaces (I'm sure there must be IDEs that solve this problem but it was no fun on vim or even notepad++)
Shift+V
}
< or >
Hit . as often as needed
Too difficult?
Not to mention you will run into exactly the same problem when copy pasting C unless you like randomly indented code
If I mix and match spaces and tabs, Python will not work and you cannot tell them apart ,it's a bit more subttle than this, but you get the idea (should you wish to).
There is no such issue with braces. This is what i meant.
As for formatting, you can use ide shorcuts to format your pasted code but it will run fine even if not formatted, not the same for Python, where it will not run.
That is true, but mixing spaces and tabs is a contrived example. Literally nobody actually does that.
Same for the point about formatting, I think. The average python dev probably spends about as much time reformatting pasted code as the average C dev spends typing braces.
I guess my point is just I think you subjectively dislike Python’s syntax (which is fine) but are trying to make objective justifications for that which seem really far fetched. Why not just say you don’t like it?
> That is true, but mixing spaces and tabs is a contrived example. Literally nobody actually does that.
Contrived or not this has happened to me when copying and pasting, but I take your point that people won't mix and match.
> I guess my point is just I think you subjectively dislike Python’s syntax (which is fine) but are trying to make objective justifications for that which seem really far fetched. Why not just say you don’t like it?
While the issues I encountered might seem contrived and far fetched to you, they have happened to me, granted I don't do Python professionally.
As I said on first comment, I don't like it because of the reasons I mentioned, some are factual,e.g. indentation when copy and pasting can mean that the script/program won't run, some are opinion-based, e.g.
they [braces] are easy to visually parse.
Two things: 1. For people not used to do programming, braces in text are usually significant - having them everywhere, and even start and end on different lines, are confusing. 2. For people not using standard US keyboard braces are often an SHIFT or ALT command, that makes it wildly different to program uninterrupted.
> 2. For people not using standard US keyboard braces are often an SHIFT or ALT command, that makes it wildly different to program uninterrupted.
I really depends on what you mean by wildly,but I've never programmed in a US keyboard (I use UK or Spanish) and it doesn't seem to be a big issue for me, clearly YMMV
The part that Python really bothers me, as someone who had write Python for almost 10 years, and right now doing it professionally, though personally I don't consider myself a Python developer rather than a Python senior user, is actually obvious to nail down, and frustratingly, difficult to handle.
SLOW. The sin of all. I don't want to make other arguments with people who just come to repeat 'Oh you can make it fast, you are doing it the wrong way'. Sorry, I have heard that 100 times, now is sounds like bug-as-feature excuse to me.
Sorry, but Python is just slow. All the stuff people might bring up is just trying everything to bypass Python itself. Multiprocessing is hell to work with, buggy and unpredictable. Cython is OK but it is not Python.
Python's slowness is like cancer, and terminally so. Down deep, it is because its overtly flexible/loose type system, the never-going-away GIL, and prematurely exposing of C-API makes it almost impossible to optimize without breaking the existing compatibility.
And most importantly, I don't think it is solvable, truth is the patches are so integrated, people already forget what problem they tried to fix in the first place and just live with it.
But it will haunt Python in long term though, people will be struggling with Python in performant production until they start reimagining a world that without Python. Maybe a better Python in that sense.
Same here. I've used Python daily or almost daily for the last ~10 years (thankfully I've used other languages too).
I've architected reliable systems on top of it but it was obvious to me almost from the get-go that Python is a terrible language. I credit my previous years of programming in Lisp that trained my mind to see through the Python mirage.
One issue is that veneer of Python user friendliness that lures people, usually those who are new to programming or don't have extensive experience with _multiple_ languages, in.
The major issue for me is that Python lacks internal consistency and is based on a inverted-design (meaning, no design really) that accumulated over the years and stinks. This makes it really easy for people to use Python and write unmaintenable code but also very hard for them to find out how to write good code. The hallmark of a terrible language is making good code hard to write.
As far as specifics, I think the module system is a joke, the packaging even worse, the REPL is not really a REPL (in the Lisp sense), ctypes is bad and has 100 different ways you can hang yourself with, there are tons of gotchas in the language and standard library that if you haven't memorized [usually by running into them], they will bite you hard, the GIL ties my hands, iterator/generator obsession makes for really bad code messes if people take it seriously and start poorly designing their APIs around it, absence of proper lexical scope, statements are not expressions and led to gross hacks like PEP 572... I could keep going for days.
I honestly think you've just worked on bad projects. Python isn't the perfect language but the quality of the available libraries makes it awesome. The individual quality of projects in any language will always vary.
My criticism is mostly aimed at the language (and its standard library) itself. The major projects I worked on, I architected from the beginning. It is when designing and architecting reliable systems that Python's terrible nature hits you in the face, especially if you're used to better languages.
Take the nonsensical but alluring Python motto "There should be one -- and preferably only one -- obvious way to do it." Assuming this was even true, which it isn't as a series of Python debacles - by its designers no less - over the years has proven, I don't want a language that shoehorns my thinking into "one"-ness. One-ness doesn't solve hard problems. I want modeling flexibility, the ability to examine a problem from multiple perspectives and explore the solution space in-parallel from multiple angles of attack. C and C++, bad as they are, let you do this. Perl lets you do that. Common Lisp is the best language I know on that front, it is _designed_ to let you do this.
I'm not going to say that Python is outright hostile to that mode of working, but it's not designed to bend / be stretched or mold with your mind in the same way that great languages are. Sooner (if you're solving fuzzy problems) or later , you'll run into its limitations and then you can't help but think that you're wasting your time.
Don't get me wrong, it is obviously your right to dislike a language.
I've designed plenty of reliable systems in Python without much issues whatsoever with making them reliable.
I have had some problems with code organization over the years (Python certainly lets you "do your own thing" which can lead to issues with organization over time) however I haven't had reliability, stability, or speed issues for 95% of the problems I've worked on.
Only times I've ever personally needed a faster language - when submitting problems in Python to online courses (algo). Then I used C#. Whatever, not a big deal :)
Python says the goal is to have "one right way" to do things, but certainly there are many, many ways to do any of those things.
> I want modeling flexibility, the ability to examine a problem from multiple perspectives and explore the solution space in-parallel from multiple angles of attack.
Can you give an example of where Python doesn't allow you to do this? I've never had this issue. Actually the opposite: mocking solutions are extremely easy with Python IMO and the tooling available (even the uber-free stuff) is awesome and easy to debug with (much easier than using Visual Studio for example - not saying VS is a bad app, it is great).
Lack of metaprogramming / code generation at runtime.
Lack of performance forcing me to severely compromise or implement things in C.
Type system is probably among the worst in the dynamic language space (compare to Common Lisp and Erlang).
Python is not an interactive programming language and can not be used to implement rapid feedback loops as is possible in Common Lisp, Smalltalk and Erlang.
The object system is very basic and error-prone compared to CLOS.
The error handling system is very limited.
I've found that all of these points translate into loss of flexibility and severely constrain the solution space. Having worked extensively in all of the alternative languages that I mentioned, there is not a single day that goes by where I'm programming in Python that I don't curse the language for being so limited / badly designed.
Agreed the GIL is annoying, but I've had no issues with creating large distributed job engines with Python. Actually the tooling made it very easy. If you're trying to do this without using existing frameworks I guess it may be difficult.
> Lack of metaprogramming / code generation at runtime.
IMO Python is the definition of metaprogramming. Actually, usually you use something that is literally called a "metaclass" to do so. Also everything is an object that you can override as you like, so not sure what you mean here. Maybe you were not aware of these features and functionality?
> Python is not an interactive programming language
That just isn't true...
Are you using libraries at all or are you avoiding them?
I've met a lot of developers who do not like to use libraries, but that is a major strength of the Python ecosystem.
I get the feeling that you are easily dismissing my points without having personally experienced the languages and ways of working I'm referring to. If that's true, then we can only agree to disagree as the only way for you to really see my point of view would be through familiarizing yourself with the paradigms and tools I mentioned.
I'll give you one example, for the last point, Python not being an interactive language. If you work in an actual interactive language (Smalltalk, Common Lisp, Erlang) you will realize what the term can mean when properly pursued and made the focus of a language, and see exactly how Python fails to live up to that.
Unfortunately I can only speak with regard to Python, but I don't mean to dismiss your views.
The language isn't perfect, but I'm sure the languages you love have flaws as well.
Regarding interactive language, can you give an example? I've tried to search and while I can find a lot of examples of what I mean in the context of Python I can't see anything that gives me good info regarding Common Lisp, Erlang, or Smalltalk.
Also, can you answer my question whether you were leveraging libraries when you used Python or trying to create all your own solutions to issues?
Python's success is richly deserved. The fact that it has been a long time coming (Python is almost 30 years old!) is an indication of the amount of sheer effort involved, and should give heart for all those working on building up the next generation of languages.
Since this forum is likely to have some folks looking towards the future, I found interesting a couple of talks by Python veteran Armin Ronacher [1, 2]. Watching that presentation makes me wonder whether I should think of this in the context of "Worse is better". That said, Python's runtime introspection is such a killer feature that it's difficult to explain how empowering it feels for someone who is relatively new to programming.
It's so refreshing to not see anyone here whining about Python forcing them to indent their code properly. The kind of people who think that's a PROBLEM instead of a FEATURE really should not be ever writing any new code, and instead should be punished by forcing them to maintain other people's code written with that same lazy misguided attitude, until they recognize how self indulgent, inconsiderate, and disrespectful that is not just to other people who have to read their code, but also to themselves.
The same kind of people criticize Lisp for having too many parenthesis, then go on to write unintelligible shell scripts, Perl and C++ code chock full of skewbald menageries of sigils and punctuation bound together with a towering inferno of precedence rules, terribly worse than Lisp's "too many parenthesis".
The best, brightest and most informed/experienced people who do not think it is a good idea for significant whitepace to be the determiner of syntactic nesting do not hold that position because they would like to inflict badly formatted code on their maintainers. You've invented a stupendously silly strawman.
That's a stupendously anonymous appeal to authority you've made there, and it's unclear what argument you're actually saying without some explicit parenthesis around all those deeply nested phrases, or some kind of grammatical diagram.
Can you please restate how we disagree in a few simple sentences? Are there any names or links or citations or concrete examples you could add to support your argument and help us respect your authority?
Speaking of best, brightest and most informed/experienced people:
Have you ever used Dave Winer's outliner based scripting language, "Frontier"? Since the syntax of the scripting language is actually a structured outline, it's impossible to indent your code improperly. It's not that white space is significant, it's that no white space or brackets or begin/end keywords are even necessary, because your code is simply an outline.
After the success of MORE (an outliner and slide renderer for the Mac), he went on to develop a scripting language whose syntax itself (for both code and data) was an outline.
Kind of like Lisp with open/close triangles instead of parens!
It had one of the most comprehensive implementation of Apple Events client and server support of any Mac application, and was really useful for automating other Mac apps, earlier and in many ways better than AppleScript.
Then XML came along, and he integrated support for XML into the outliner and programming language, and used Frontier to build "Aretha", "Manila", and "Radio Userland".
He used Frontier to build a pioneering fully programmable content management system, blogging and podcasting platform, with a dynamic HTTP server, a static HTML generator, structured XML editing, RSS publication and syndication, XML-RPC client and server, OPML import and export, and much more.
He basically invented and pioneered outliners, RSS, OPML, XML-RPC, blogging and podcasting along the way.
Frontier certainly had its problems, but it was enormously successful. I think it's a good idea to learn from Dave Winer's experiences successfully designing and using practical powerful scripting languages.
>UserLand's first product release of April 1989 was UserLand IPC, a developer tool for interprocess communication that was intended to evolve into a cross-platform RPC tool. In January 1992 UserLand released version 1.0 of Frontier, a scripting environment for the Macintosh which included an object database and a scripting language named UserTalk. At the time of its original release, Frontier was the only system-level scripting environment for the Macintosh, but Apple was working on its own scripting language, AppleScript, and started bundling it with the MacOS 7 system software. As a consequence, most Macintosh scripting work came to be done in the less powerful, but free, scripting language provided by Apple.
>UserLand responded to Applescript by re-positioning Frontier as a Web development environment, distributing the software free of charge with the "Aretha" release of May 1995. In late 1996, Frontier 4.1 had become "an integrated development environment that lends itself to the creation and maintenance of Web sites and management of Web pages sans much busywork," and by the time Frontier 4.2 was released in January 1997, the software was firmly established in the realms of website management and CGI scripting, allowing users to "taste the power of large-scale database publishing with free software."
I just don't think Winer should be used as an authority for anything like this. What he pioneered was "just make it work for the user" solutions from a usability perspective, and not from an engineering perspective.
Winer part in the XML-RPC and SOAP outcomes was a source of misery for developers in the early part of this century. MORE was neat because it had a place on the Macintosh System 6 platform, when there was literally nothing else of its kind. What did MORE evolve into later? Nothing.
Winer has long been an advocate of "the semantic Web", which is a user-focused idea of what's essentially an engineering problem. The semantic Web was/is a dead-end and in its time it led to a great many stupid uses of HTTP.
Winer can be celebrated as an advocate for users but not as a great engineer.
As much of a fan of Python's syntax as I am, there are real problems with whitespace-sensitive syntax.
It's not that it's hard to parse or necessarily to write.
The trouble is that generic tooling around text is really bad, especially various websites and chat boards still haven't figured it out.
Code is a way to share ideas, which was one of GvR's key insights. People share a ton of code in discussions and such and WSS makes it a bear; even here I have to tediously type in four spaces to share a snippet of Python.
> The kind of people who think that's a PROBLEM instead of a FEATURE really should not be ever writing any new code, and instead should be punished by forcing them to maintain other people's code written with that same lazy misguided attitude, until they recognize how self indulgent, inconsiderate, and disrespectful that is not just to other people who have to read their code, but also to themselves.
Having gotten worked up like this myself plenty of times, I think you're treating someone doing a bad job as an injustice to you.
Unless there's a malign intent, it's simply not. And elevating it to an injustice is creating stress and strife for yourself (which you're venting here) that you don't need.
I consider it a must that code should be correctly indented, no matter the language. What I have issue with is relying on whitespace to represent scope etc. With .sass files I've had too many occasions where moving the content around between editors has completely borked the file, and without braces it's tedious to correct the indentation. This is mostly the fault of text editors trying to be "smart" about indentation, and failing miserably, but it's a pain I'd rather not exist.
In over 12 years, as mostly a C# developer, I've only seen one person not use the standard indentation style and this was because he was a junior who didn't know that you could format the code with a shortcut.
edit:
I should point out that it was always rejected at PR time, it just took me a few PRs to realise what the problem was
You’re spot on… As someone who uses Node/JS for most of my projects, I was actually surprised at how easy math is in python. I was given a homework assignment with the ability to use whatever language I wanted, so I picked JS. After hours wasted trying to do math with large numbers, I switched to python because others seemed to have success with it. Needless to say, I finished the assignment an hour or so after that. (Note, I would still use Node over python for an API any day of the week, personal preference)
Yes, Yes, Yes. I remember when I translated Bitcoin Python code to JavaScript, I was frustrated. Operator overloading is not available in JavaScript. By default JavaScript only supports 32 bit number. You have to use BigInt or use 3rd party library like big-integer. There is no default support for module exponentiation. And many more.
Consider that 10 years ago Ruby and Python used to be spoken about as a pair, with Ruby being the more popular one. Now Ruby is rarely heard about, JavaScript has eaten it whole and only the bones remain.
What stood out to me, reading the comments, is how among the first few comments I read there were 4 comments from people with decades of experience writing python professionally. Each stating their dislike for the language and citing entirely different parts of the language as its biggest problem.
Personally, I'm primarily a C++ programmer with about a decade of professional experience in C++. I also used to do daily work in C# for a few years. For me, python is a language which I use for making simple command line tools, for processing input and clobbering together different tools. I use it to build interfaces which streamline certain workflows. Essentially, I use it as a more powerful replacement for bash or batch. For that purpose, I find that the language works wonders and is quick and easy to work with.
The problem is that when they did the big Python 3 transition they didn't fix the real pain points (with the exception of string types), many of the things they fixed were not a problem, or worse they removed things that were actually useful. Such as the infamous print statement.
So now they can never fix the real problems, because everybody is so traumatised by the Python 3 transition.
I'll also say that the rate of development of Python is incredibly slow, slower than even JavaScript's TC39 if that's possible. I waited my whole Python career to get format string interpolation.
Python is not eating the world so much as becoming the new Excel. It has a pretty easy learning curve for people who are not programmers. It has pretty good built in tooling. A domain expert, who is not a programmer, can put together pretty impressive models and visualizations relatively quickly.
However, like with Excel, there are serious issues regarding reproducibility, performance, and long term maintenance.
That's all good and fine until you want this model you built with tooling you barely understand to be deployed. Then your gross data scientist spaghetti code gets handed off to some one who actually has the skills to turn it into a robust deployable code base.
I love that Python has brought a lot of people into programming and that it makes it relatively easy for domain experts to do advanced analytics. However, another side of me wishes that there were a less user friendly language being used for data analysis so that it would force all of these people to actually learn to code half-way decently.
For what it's worth, I have none of the issues the currently top voted comments describe. Dependency hell (apt does that for me), closer to burnout any time I work with it, would rather work with C++ (a language with a completely different purpose)? If the author of the article had written this, people would fall over themselves to say how wrong they are. Nearly everyone who tried Python loves it. From this thread you'd think we're talking about Visual Basic for Applications.
There is a reason Python is one of the most popular languages. PHP was popular because it's quick and dirty, but Python is popular because people love working with it. The arguments in this thread are beyond me. (They're also unsubstantiated, but if I ask for substantiation I'm sure someone can rationalize some reasons to dislike any language, so that's pointless.)
One thing very interesting about Python is how gradual the ride up has been. Perl dominated then sputtered. Java shot up and hit the ceiling. C# is mostly limited to Windows, JavaScript mostly to the web. Ruby stole the spotlight for a few years and then faltered mysteriously. Others found their niche. But Python just keeps chugging along slowly gaining ground almost everywhere like a tortoise to multiple hyped-up hares.
Am I correct when I assume that most of the people hating on Python here are using it for medium or big projects?
I feel like the dev community figured this out ages ago:
Statically typed languages are easier to read, harder to write, dynamically typed languages are easier to write, harder to read (and also harder to refactor).
For small projects (scripts, prototypes, etc), you don't care about readability, you just want it to be as easy to write as possible. Python really shines here.
For bigger projects, using Python is painful, but that's also the case with pretty much any other dynamic language..
Considering Python can power the largest web sites (Netflix, Instagram), I don’t understand why there’s so much use of much more complicated platform/languages (eg java, .net)?
Note: I am an amateur which is likely the reason for my ignorance.
A lot of stuff powered "by Python" is in practice powered by multi-language infrastructures, which can be pretty complex. For example, the Python ecosystem for data science is mostly implemented under the hood in a mix of C, C++, and Fortran, with high-level Python bindings, due to a mixture of legacy reasons and pure Python being too slow. Obviously lots of people still like the end result, but saying this kind of software is in Python is only half the story.
- Language design characteristics like static typing and a general lack of support for "magic" can help maintain order in large, long-term, many-contributor codebases.
- The JVM and CLR are both significantly faster and less resource intensive than CPython for a lot of workloads.
- Java and .NET library ecosystems are mature and extensive. Python 2's ecosystem was also mature and extensive, but the community is in the process of throwing it away.
- The real heavy lifting for media transcoding, storage, and serving is certainly not done in Python. Python is acting as orchestration glue for C/C++ programs there, which is a role it does well.
Someone I know made the observation that Python doesn't scale well with code complexity and team size. Statically tyed languages like Java give you a lot of guardrails, and IDEs make it easy to navigate around code, so it's easier for larger teams with larger projects to safely reason about things.
Difficult problems remain difficult no matter what language you use.
Also, there's a real cost to everything being dynamically typed in large systems. Humans can't keep all the program types in their brain, and having the compiler help you out is kind of a big deal.
Java and .NET are no more or less "complicated" than Python or PHP or any other language/stack. The most suitable language for any app/service is what the founding team knows very well.
You can do anything in anything. Everything boils down to trade offs for the problem or company at hand.
I can say with Java and .NET shops, it's usually because the company has heavily invested in Java and .NET infrastructure, process, automation, tooling, monitoring, etc. If there are certifications involved, I also know for a fact that Microsoft feeds big discounts on expensive products (SQL Server) depending on how many employees you have with Microsoft Certifications.
It becomes a self feeding system. You're already invested so much in the stack that the choice of using something else is far more complicated than language comparisons. It's one of the reasons that there are so many language options available that run on the JVM that can take advantage of all of that investment without forcing the devs to write raw Java (Groovy, Scala, Clojure, Kotlin, etc).
It isn't just the company's heavy investment, it is the community's.
The availability of good, tested, performant OSS libraries available in Java is unparalelled.
People talk about languages like Python being "faster" to develop in because they're easier to write. This is true until you need a solid concurrency friendly loading cache or a DB connection pool that works at scale and in Java that's just adding a dependency that you know you can trust.
Python is pretty slow. If you can rewrite your stuff in Java or Go or something else that runs faster you can save half your servers at least. At a certain size that's a lot of money.
> I don’t understand why there’s so much use of much more complicated platform/languages (eg java, .net)?
What language is Python implemented in? Is it Python, or something else? This tells you that even the implementors of Python don't think Python is suited for everything.
No it doesn't. As someone else mentioned, RPython exists, and in general, bootstrapping is nice, but there are a variety of compelling reasons to not rewrite CPython in python, that have nothing to do with it being "suited" or not.
History, reliability, simplicity, lack-of-value, etc. At this point there's not much to gain by rewriting cpython in python (pleasing some guy on the internet), and quite a lot to lose (simple c-interop) for what would be years of work by a lot of people.
It wouldn't have been years of work if they started right after Python was first implemented instead of a big-bang now. For 25 years each time they've implemented a new feature they've chosen C over Python to do it, for reasons of it being better suited, some of which you've listed yourself! Like simple C-interop!
I don't think it's a criticism of Python! I've worked on meta-circular implementations of languages and I don't do it for the sake of it. Not every language is suited for every task.
Right, but all of those are historical, not innate, which makes your comment wildly misleading.
If the Python ecosystem weren't c-based, it wouldn't matter. But it is. But that has nothing to do with python the language in a vacuum, and everything to do with choices some guy made 25 years ago.
You typically have to bootstrap your language in a different one. There are some languages that have since gone back and re-wrote the compiler or runtime engine in the same language (in those languages that need a runtime).
I would not use Eve-Online as an example of performance.
Once it gets to a few thousand concurrent users they have to drastically slow down the server tick rate. It's like playing a turn-based strategy game at that point.
There was a time (more than a decade ago) when I wondered if I should learn Perl or Python. Unfortunately I picked Perl, subsequently forgot it and learned Python instead.
Ruby is a nicer language for general scripting, ie “better bash,” as that’s what it was originally designed for. It’s also a bit faster, and has a vastly better / simpler package management system.
Python does have great scientific / math libraries, and is worth knowing.
One nice thing is the two languages are so similar that it’s pretty easy to switch between them and use whichever fits the bill.
I was a Ruby programmer for a long time, but I think the world has moved on. Nowadays, I want a language with some type checking, and Ruby is extremely hostile to any kind of static analysis. Rails was pretty exciting as a replacement for how we built websites in the perl/php era, but now that React has taken over, Rails feels weirdly bloated. I would probably choose Typescript for basically everything now.
In my opinion, the biggest problem facing Python is still concurrency. The mitigation efforts still look to me like Perl's 'bolted on' implementation of object orientation. Python should have gotten rid of the GIL during the 2 to 3 transition. It may have also been an opportune moment to introduce optional typing and maybe even optional manual memory management, making it useful to develop almost all kinds of software. Write a WASM backend and it could have even been used for front end development.
There have been attempts to replace the GIL with locks on individual lists, dictionaries etc but they came out a wash performance wise. Perhaps this can change with gc.freeze and the new dictionary structure.
This makes feel a bit sad/somber, because a lot of people are missing out on the exhilarating power of using advanced powerful type systems. I wish the ML community had picked up a few notes from Python, and built an Ocaml-like language with beginner-friendly Pythonic syntax (Reason comes close). I wrote about this here in the past (https://news.ycombinator.com/item?id=15549101); quoting myself below:
I'm a big fan of strong static type systems. I believe type-safety increases code quality significantly.
I used to think several years ago, that the main benefit of strong static typing was code safety / eliminating a whole class of bugs. But I've changed my opinion. I now think the biggest benefit is that it makes the code a lot easier for other people to read and understand.
I mean I have multiple personal projects where I've used Python (which is a dynamically typed language), but these are small one-off projects. But I think when working in a team, especially a large team, having types becomes a huge thing. Having types for objects is especially useful. Having types forces you to think more clearly about the structure of your data.
It's really sad when I see `foo(bar)`, and I have no idea what the type of `bar` is, and if it's an object, I have no idea what fields `bar` has. I have to simply guess the structure of the various implicit types by looking at the code (sigh). It makes the code difficult to read, and rather unpleasant to work on. Not to mention, all the multitude of bugs that come from duck/dynamic typing.
I don't think good statically typed languages are hard to use at all. Type inference has spread everywhere that the old argument of having to repeat your types doesn't hold anymore. TypeScript, Flow (JavaScrpt), Haskell, languages from the ML family are really good at type inference. Even the `auto` type inference in C++17 was better than I'd expected.
If dynamic typing continues to grow in popularity, I don't want to even begin to fathom the number of large projects that are going to built, which will suffer from a avalanche of TypeError-class bugs that could have easily been avoided with static typing. One encouraging sign I've seen in the Python world is that optional static typing is reluctantly being adopted in many circles. There was mypy for a while, and now, there's a new type checker called Pyre (written in Ocaml). I find hope in these little things.
I've been told that my choice of python was poor numerous times. 'Do java/.net' etc. is something I've heard all throughout my career. I use python because it doesn't work against me - It's not the best language but I think its definitely the most versatile. I love python and it will be around for another decade I guarantee it.
Ease of use and advanced features are not polar opposites.
I do a lot of geospatial work and Python bindings for GDAL [1] are godsend. In 10 lines of code, I can read shapes stored in different projections and file formats; load, reproject and merge them into an in-memory SQLite database; do all sorts of complex geospatial operations on them using spatial extensions, and save results to a file.
I'd spend half a day writing C++ code that Python abstracts away into very few commands. It's a joy to work with, and I struggle to see any downsides.
WRT to Python at least, the skill range is more impressive than the low bar. High-skill programmers can do some pretty advanced things with it, and those advanced things can still be easy to use by lower skilled programmers.
I've come back from Ruby as the default scripting language to Python. Over the years, the accumulated Python scripts are much more consistent and easier to read than aged Ruby script. Ease of use, sure. But the language itself I would argue is much better thought-out and consistent. Now, only if it has better type and dependency management ...
The number of views of questions on Stack Overflow is a bad metric to measure language popularity.
This metric combines popularity and complexity/unintuitiveness.
If a language is half as popular but twice as complex (or unintuitive) as another language, then it will get the same number of questions and same number of views.
If the language is becoming more fragmented; more different kinds of frameworks and tools then it will translate to more questions on stack overflow.
Fragmentation of the language is not necessarily a good (or bad) thing but it has nothing to do with the popularity of the language.
But definitely the JavaScript ecosystem is pretty homogeneous at the moment (ReactJS or VueJS on the front end and Node.js on the backend).
If you hate python, just write whatever your hearts desires. There is no use in saying "oh python does not let me know when a variable I expect to be list is string", well yes, it is a dynamically typed language, please learn the difference between dynamically and statically typed languages before hating on any language. As a bonus, you can check if a value is list or string and throw your own error, python is versatile, although it probably would be an antipattern and you should go back to writing C.
If you love the good old days of Fortran, then I am sure many Financial Institutions and Aerospace industry or old school mathematicians would love to have you.
I would argue that Pythons biggest issue is performance, which translates in large applications directly into costs. You will need more machines to scale and you will need more power per operation compared to other programming languages.
I think Python makes sense for prototyping and stuff like Jupyter Notebooks. But once mature and going into production I recommend porting stuff to something faster like Java or even Javascript nowadays.
While I use more than just Python in my work, it remains my go-to toolset for a range of tasks. For instance, Python offers the best interactive programming experience of any language today. You can step through code flow and inspect values with great ease. This is particularly valuable during prototyping, as you are getting a handle on interactions between your application and third party dependencies. Further, as designs take shape, you can quickly assemble a proof of concept and learn how to address a problem or design. Python can be used with little or any inheritance. Adopt a compositional style. You can assign types. You don't need to use all of its bells and whistles if you don't want to. This allows you to use Python in a way that streamlines it for porting to a strongly typed, high performance language such as Rust, should the need or desire to do so ever arise.
The Python community is truly global, hosting local events in hundreds of countries. So many talented, helpful people have been dedicated to the language and community over a very long period of time. Look at its ecosystem on GitHub to get an idea of how that manifests. Questions about how to solve a problem are often a single search away from being answered on Stack Overflow, Google groups and countless channels on Freenode IRC. Python conferences are a great place to learn and connect.
I like a lot of things in python, but unfortunately, everytime I happen to use anything which uses python, there's a good chance it won't work on my environment. And so I'm very surprised how python can be so successful if it doesn't actually work. I literally fix my env for some tool A, only to make it incompatible with tool B. And I can't believe I'm alone.
The comment "Meanwhile, the Unix shell had different problems -- it offered a suite of utilities for common tasks, but ran so slowly it couldn't handle complex logic."
This doesn't make sense and also why the F is the " Dutch national research centre for math and computer science. " using c and nor Fortran
Interesting article, I would love to read one regarding Java and Kotlin, or if even someone here would share some knowledge with me, I would be grateful.
Is Java dying, what is its future?
Where is Kotlin heading, currently it is the hottest language for Android development, can it also compete with Python in some branches?
Java moved to a six month release cycle and it has breathed new life into the ecosystem as a whole. Its one of the most robust and important languages out there.
Java will likely never truly die because of legacy applications. It's also still firmly set in the market and that is not looking likely to change in the future. Its also the king of microservices with Spring Boot.
Kotlin could well take over the mobile space. Many of the touted benefits of Kotlin are slowly being added to Java itself.
Java itself may decline as a language, but I'm near certain that the JVM will be with humanity forever. Someone will be programming in a JVM language somewhere. Whether it's some future language or Java 532.
I recently gave a Python book to a co-worker that wanted to experiment with deep learning.
Co-worker: Wow - this book is like 900 pages long!Me: Don't worry, it's mostly whitespace.
And that is one of my favorite features of Python - easy to trace and code!
"I was very disappointed in how the people who disagreed technically went to social media and started ranting that the decision process was broken, or that I was making a grave mistake. I felt attacked behind my back,"
I don't know the answer to that question, but this post from February 10, 2006 captures some of Guido's frustration with the way certain contributors would insist on changing Python in ways in which they might be able to win an argument about syntactic disambiguousity and technical implementability, but at the expense of that ineffable sense of essential Pythonicity.
An incident on python-dev today made me appreciate (again) that there's more to language design than puzzle-solving. A ramble on the nature of Pythonicity, culminating in a comparison of language design to user interface design.
Some people seem to think that language design is just like solving a puzzle. Given a set of requirements they systematically search the solution space for a match, and when they find one, they claim to have the perfect language feature, as if they've solved a Sudoku puzzle. For example, today someone claimed to have solved the problem of the multi-statement lambda.
But such solutions often lack "Pythonicity" -- that elusive trait of a good Python feature. It's impossible to express Pythonicity as a hard constraint. Even the Zen of Python doesn't translate into a simple test of Pythonicity.
In the example above, it's easy to find the Achilles heel of the proposed solution: the double colon, while indeed syntactically unambiguous (one of the "puzzle constraints"), is completely arbitrary and doesn't resemble anything else in Python. A double colon occurs in one other place, but there it's part of the slice syntax, where a[::] is simply a degenerate case of the extended slice notation a[start:stop:step] with start, stop and step all omitted. But that's not analogous at all to the proposal's lambda <args>::<suite>. There's also no analogy to the use of :: in other languages -- in C++ (and Perl) it's a scoping operator.
And still that's not why I rejected this proposal. If the double colon is unpythonic, perhaps a solution could be found that uses a single colon and is still backwards compatible (the other big constraint looming big for Pythonic Puzzle solvers). I actually have one in mind: if there's text after the colon, it's a backwards-compatible expression lambda; if there's a newline, it's a multi-line lambda; the rest of the proposal can remain unchanged. Presto, QED, voila, etcetera.
But I'm rejecting that too, because in the end (and this is where I admit to unintentionally misleading the submitter) I find any solution unacceptable that embeds an indentation-based block in the middle of an expression. Since I find alternative syntax for statement grouping (e.g. braces or begin/end keywords) equally unacceptable, this pretty much makes a multi-line lambda an unsolvable puzzle.
And I like it that way! In a sense, the reason I went to considerable length describing the problems of embedding an indented block in an expression (thereby accidentally laying the bait) was that I wanted to convey the sense that the problem was unsolvable. I should have known my geek audience better and expected someone to solve it. :-)
The unspoken, right brain constraint here is that the complexity introduced by a solution to a design problem must be somehow proportional to the problem's importance. In my mind, the inability of lambda to contain a print statement or a while-loop etc. is only a minor flaw; after all instead of a lambda you can just use a named function nested in the current scope.
But the complexity of any proposed solution for this puzzle is immense, to me: it requires the parser (or more precisely, the lexer) to be able to switch back and forth between indent-sensitive and indent-insensitive modes, keeping a stack of previous modes and indentation level. Technically that can all be solved (there's already a stack of indentation levels that could be generalized). But none of that takes away my gut feeling that it is all an elaborate Rube Goldberg contraption.
Mathematicians don't mind these -- a proof is a proof is a proof, no matter whether it contains 2 or 2000 steps, or requires an infinite-dimensional space to prove something about integers. Sometimes, the software equivalent is acceptable as well, based on the theory that the end justifies the means. Some of Google's amazing accomplishments have this nature inside, even though we do our very best to make it appear simple.
And there's the rub: there's no way to make a Rube Goldberg language feature appear simple. Features of a programming language, whether syntactic or semantic, are all part of the language's user interface. And a user interface can handle only so much complexity or it becomes unusable. This is also the reason why Python will never have continuations, and even why I'm uninterested in optimizing tail recursion. But that's for another installment.
This is wrong in terms of the web space. With the modernization of PHP and the emergence of Laravel, and continued evolution of Drupal and Wordpress, I don't think PHP has ever been this dominant in the web stack. Django is an also-ran.
I love Python but.... I would not go to the lengths that this article proposes. The dev world recreates itself every year with dozens of new high level languages and task specific support libraries. Every developer has their favorite language (Everyone is comfortable with their personal preference) That being said, I enjoy Python and it provides a quick, easy script to accomplish task in my space but I would not go as far as the article in saying that it is the end all be all for everyone. Use what you like, and understand that all languages have strength and weaknesses for task.
I find Python to be a great alternative to writing bash scripts. I did that with this semi-popular Android tool (https://github.com/ashishb/adb-enhanced). Python is close enough to be a language but straight-forward enough to be a shell script.
For the Python haters here. No, Python is not as good as GoLang or Kotlin for maintenance but IMHO, it is better Javascript/Typescript/C++ for writing non-performance critical code.
What I find baffling is that almost every program I'm interested in as a creative power user has a Python API for scripting, from type and graphic design to 3D modeling, rendering and music making.
I'm a bit into web design/frontend as well, and was thinking about either learning JS and Node more deeply, or add some other "backend language" like PHP or Ruby, and learn their frameworks like Laravel or Rails.
Should I pick Python, Django or Flask instead to cross the gap to my creative hobbies?
The data synergy of Node + Web is hard to beat, and JS is very elegant for writing asynchronous code. Also, I've found the web (canvas, svg, webgl with Three.js) to be an amazingly hackable creative platform. Just my two cents.
I have been a Python guy for 10 years now switching from another profession - I was a mechanical engineer. I tried to learn other languages like Java ,C and was partially successful too. But I don't remember creating anything useful in those languages.
Everything was so verbose that I got drowned in the syntax rather than the logic behind the program. For me, Python signifies ease of development and fast iterations and as such is suited for any young developer.
A joy to write and read. But a pain to debug and maintain. Coded a big project in Python once. When the project was still small everything was perfect. When the project grew it became unsustainable. Of course I was an already experienced C# dev, just getting started with Python but the problem is there. A whole lot of errors spilling to runtime is a big pain. But that's a problem with every dynamic language at variable levels.
Tools, tests, and logging are the cure to those troubles. I've not experienced substantial problems with projects large or small.
If a project is known to be huge up front and/or some developers will be inexperienced choosing a language with additional guardrails would be prudent.
I've not written that many heavy applications in Python (Java mostly). I do like the dynamic nature of it. I find myself leaning more towars dynamic these days, appreciating Javascript today as well. I like the v.env side of Python. Many languages have this plumbing to get a project started and Python is the most comfortable of them all. Python is smart. Python is easy to teach also.
Suppose there are more things in the programming universe than are strictly represented in (say, R5RS) Scheme. Because Scheme has things like continuations and syntax-transformations, we can put them into the language.
It's easy to get going on it, it's productive for writing scripts and experiments. It's dreadful for programming in the large, but only in comparison to languages designed to take account of the lessons learned about large code bases and team working. Most programmers have never encountered these and have very little idea of what they are missing.
Can totally agree that python is eating the world, while in the same time being too opinionated on certain things and lacking in certain points. I would rather see lua more adoption - its the language designed right - minimal and logical, easy to comprehend quickly for anyone who got any exposure to other C-like languages. Easy integrations as a bonus.
Python’s killer feature - and perhaps reason for its popularity- seems to be it’s nice-looking syntax, which is only the most superficial of qualities. Below that surface, the semantics and its standard libraries are all very irregular, and always leave a bad taste in my mouth. I now avoid touching it if i can.
I have used all code from c+, java, php, javascript, ruby and more and found python is the easiest and simplest to use (you do not have to setup 5 classes to declare 1 variable) the only thing i hate about python is there is no built in parser to handle formatting.
Javascript might be eating the world of web development, but there are a whole lot of other scenarios outside of web programming that Python is more likely to be used in, IMHO.
Saying this again, is this a cover effect ? I'm strongly thinking that python reign just peaked, and that either python will change drastically into a more solid static core or people will go Julia..
Never had the chance to use python but what I hear most about Python is that it's easy to develop/saves developer time. Is it still true when maintaining a project vs developing from scratch?
Wow. uBlock registers at least 50 unique blocks, and my laptop still churns trying to load all the extraneous ad content ZDNet tries to jam down the pipe. ZDN stopped being journalism long ago
Probably a kind of premature optimization on my part, but this is why I chose Golang over Python as my primary language. Maybe one day Python will have its own compiler without having to drop into writing C, who knows? For know I think the features of Go are just more attractive than those of Python.
Python has a compiler without dropping into C (cython).. but it's mostly used these days for glue code between c and Python, or for small bits of code that need extra optimization.
I do most of my programming in Cython these days, mainly maths/graphics-related, but all kinds of stuff. It's great for that! I do the intensive calculation parts with typed C variables, structs, arrays, at C speeds, while doing the once-only stuff like loading and saving images in pure Python. I guess it's weird floating between C and Python from line to line in a program, but I like it a lot. (Not having to type semicolons and curly braces anywhere is no small benefit also.)
I use pythonanywhere. Seems to work well enough and isn’t expensive. Main problem I have with it is I cannot use Altair-viz in a Jupyter notebook because their notebook server is out of date.
Some good news for you: Pythonista absolutely supports numpy, I followed advice on the web a few months ago to install it, and I just tried it to make sure it still works.
It’s really terrible that python should be growing. It’s a bad language and a bad ecosystem ...
The knowledge exists that we shouldn’t be encouraging systems to be built with the patterns/tools commonly advocated in the python world — at what point do we acknowledge that the forces promoting this kind of ecosystem are essentially propaganda-like mind viruses ... so much future suffering that will not be prevented ...
Well, I could be put in the ‘Python hater’ category: I do love Python for short programs using TensorFlow, the SpaCy NLP library, PyTorch, etc. I really hate using Python for large applications. Everyone gets to choose their own favorite languages, and for me that would be Common Lisp, Haskell, Java, or Scala for working on large projects. Each to their own though.
When it comes to raw performance, PHP7 is much faster than Python.
PHP is more flexible when it comes to building web applications. But in some areas Python just have better libraries (computer vision, matrices, high-level mathematical functions). I remember reading that PHP is going to add FFI and I believe FFI will have a big impact in PHP world.
I really hoped Julia would become a major player in scientific computing and data science, but it looks like ever less probable. I like Python, but for science I like Julia more.
>"In the past, it had always been clear that if there were a decision to be made about a change in the language or an improved feature, a whole bunch of core developers would discuss the pros and cons of the thing. Either a clear consensus would appear or, if it was not so clear, I would mull it over in my head and decide one way or another. With PEP572, even though it was clearly controversial, I chose 'Yes, I want to do this', and people didn't agree to disagree.
This is a characteristically practical, effective Dutch attitude, known as the "Polder Model". The United States currently suffers from the opposite problem of extreme partisanship and gridlock.
The polder model (Dutch: poldermodel) is consensus decision-making, based on the acclaimed Dutch version of consensus-based economic and social policy making in the 1980s and 1990s.
The polder model has been described as "a pragmatic recognition of pluriformity" and "cooperation despite differences". It is thought that the Dutch politician Ina Brouwer was the first to use the term poldermodel, in her 1990 article "Het socialisme als poldermodel?" (Socialism as polder model?), although it is uncertain whether she coined the term or simply seems to have been the first to write it down.
[...] Other uses
The term polder model and especially the verb polderen (to polder) has been used pejoratively by some politicians to describe the slow decision-making process where all parties have to be heard. The model flourished under the "Purple" governments of Dutch prime minister Wim Kok, a coalition including the traditional rivals the Labour Party (a social-democratic party, whose colour is red) and the People's Party for Freedom and Democracy (right-wing liberals, whose colour is blue). In the declining economic climate of the early 21st century the model came under fierce attack particularly from right-wing politicians and Pim Fortuyn in his book entitled De puinhopen van acht jaar Paars ("The wreckage of eight years Purple").
Historical background
[...] A third explanation refers to a unique aspect of the Netherlands, that it consists in large part of polders, land reclaimed from the sea, which requires constant pumping and maintenance of the dykes. So ever since the Middle Ages, when the process of land reclamation began, different societies living in the same polder have been forced to cooperate because without unanimous agreement on shared responsibility for maintenance of the dykes and pumping stations, the polders would have flooded and everyone would have suffered. Crucially, even when different cities in the same polder were at war, they still had to cooperate in this respect. This is thought to have taught the Dutch to set aside differences for a greater purpose.
What stood out to me: online bullying (dressed-up sometimes as concern/outrage) is what led to Guido stepping down.
Python is great, but it will never be perfect. But the outrage stuff, the entitlement stuff, essentially anything that happens on Twitter... Python seems like a victim of it's own success. Once it got popular, it couldn't scale the checks that might have stopped this runaway train of shit-tweets and bickering on the mailing list.
Personally I don't like a lot of the things that have happened in 3 (asyncio, type whispering, fstrings). I can't help but wonder if, again, this is due to becoming too popular too fast. I say that because aesthetically these features seem to go against the Zen of python.
May you never have users, but if you do, may they not have Twitter accounts!
What’s wrong with fstrings? I use JavaScript at my day job and leaning python for Raspberry PI projects. Fstrings looks like it’s better than .format or + concatenating + a bunch is strings. Especially large strings.
I dunno, R is also exploding in popularity but it seems to be able to manage itself really well. To the point of successfully creating an environment that attracts women and other underrepresented groups towards it.
Switch to JS was like obtaining new home, don't want to write Python code anymore. Also "Eating the World" isn't correct anymore, 'cause Python popularity decreasing last ~5 years according to Github stats.
It appears to consume over 17x more power than JavaScript, 40x more power than Java or 75x more than C/Rust. The only language which performed worse than it in terms of pure power consumption was Perl.
While I'm sure these results aren't exactly indicative of day-to-day usage as they're based off of benchmarks which are not exactly representative of day to day use cases it nevertheless raises the question of whether Python is an appropriate choice of language where the application itself is likely to see a high level of total use. I suspect the answer is "more work needs to be done" on the subject to truly assess the impact and a big part of the answer is "depends on how you're using it". It'd be good to see some standardised tests of web frameworks. For instance, load testing a standardised server side version of the TodoMVC project to determine what happens for each framework there, along with what happens when you stick a cache in front of them and so on.
It'd also be interesting to attempt to estimate how much power consumption and hence CO2 generation each given programming language is responsible for given their marketshare and typical usage profiles - although I suspect in practice this would be quite difficult given the diverse usage profiles of modern programming languages. And obviously the fact that machine learning libraries, for instance, will naturally be quite energy intensive - it's probably the case that Python actually does quite well there because most of its popular libraries for that are just wrapping C or similar rather than being pure Python code.
Given the success of V8 and PHP7 in terms of boosting the performance and presumably energy efficiency profile of their respective languages it'd be nice to see the mainline CPython interpreter undertaking a similar kind of transformation. Obviously we have things like PyPy but it seems to me the Python community needs to be united around a solution to this which forks simply are not going to be able to drive.
I feel like the world has gone insane when it comes to Python. Or have I gone insane?
Python is an incredible waste of time. No types, IDEs that suck (a side effect of not having types), very partial validation with linting tools that are an afterthought to the language, indentation is a mess (good luck moving files between different editors and environments or through clipboards), slow, agonizing deployment (pyenv? Come on).
I’ve been doing C# these past two years and it is laughably effective in comparison, almost like cheating. Standard libraries covering almost anything, tooling that couldn’t get any better, runs everywhere, lighting fast builds.
Most importantly, in C#, due to the incredible tooling and language design, if it’s built successfully then there’s a very high chance it’ll do what I want it to right out of the bat. In C/C++, much less so. In Python it’s almost guaranteed not to do anything but crash unless I’ve ran everything through in depth coverage testing and I’d still take the C# build checking every time.
My Python is pretty good, and I’ve known it for more than 15 years now. I’m not taking this position out of ignorance.
For the love of god, go learn a proper language. It can be C#, which I think is as perfect platform/language as anything, or Rust or Go or even Java.
I see young devs on their vim or Sublime editors developing Python and I want to shout to them, that’s not how it’s done. You’re missing out on decades of industry evolution and know how in things like the Visual Stdio and .NET or even Java and JetBrains/Eclipse. You’re using stones to start fires where you could just use an oven.
I still use Python, but it’s always for very short and specific programs. It’s just a bad tool for more than that. If you are unable to resist the temptation of using Python for that short program, just make sure it never ever has a chance to grow.
Regrettably the ML world lives on Python, fine; let them have it.
I hope that Julia eventually makes such a hit that performance gets taken more seriously as part of Python devs (yeah there is PyPy, but gets seldom used in the field).
I am like you though, my introduction to Python was version 1.6, and due to my experience with its performance, I never used it for anything beyond shell scripting on steroids.
As proven by Smalltalk, Dylan, Common Lisp and JavaScript, dynamism doesn't need to be slow.
Python’s lack of typing means the IDE heavily uses heuristics. It’s nowhere near as good as Intellisense on C#. Take refactoring a variable name as an example.
I've only ever really used intellij and pycharm, so I don't know what intellisense can offer for refactoring a variable name, but I'm curious. Can you explain what you mean?
The "heuristics" are down to Python being _dynamically_ typed, not it's lack of typing, so the type can't be known until run-time. Python is actually _strongly_ typed.
And that's a sad thing to hear. Before my death by thousand downvotes, I'd like to tell you why do I feel that way.
Back in the day when I was way more inexperienced than I'm now, I was a die-hard Python fanboy. To me, it seemed like a best option available due to it being way more concise and readable than JS/PHP, not to mention that it was way more powerful and dense. I didn't really bother with type safety at the time and I wasn't exposed to any real criticism of it at the time, as it happens often when you're just too new, and I was more than fine with it.
Time mercilessly marched forward since then. I worked with different codebases, changing tech stacks and languages a bit, learning new stuff and so on. With some experience, lots of self-education and a vast amount of different workflows to compare with, I've realized one important thing. While Python may be better than some languages that are considered trash by industry consensus, it's also a trashcan language to ones with sound and flexible type systems, not to mention that, across dynamically typed contenders, it's nowhere near as powerful as literally any solid Lisp.
That said, I feel bad for the industry that "dives into Python" more and more over the last years. While it may seem like a pretty easy and cool-ish language to write in, you will inevitably have large problems at scale, it just doesn't have what it takes to be easily scalable, and most importantly, it's just tedious to refactor when it's big, even though the language itself is mind-numbingly easy and even your grandma can jump right in.
In a conclusion I'd like to say that you'll be better off without using Python as your project's main driver. Yes, it's simple to write it and the developers supply is abundant. It's easy to start. It'll just be a major pain to scale it and support it later on.
Python has a more powerful type system than any mainstream language except for C++ (templates). No other mainstream language has support for explicit marking of variance of arguments, nor does any other language support literal (dependent) types, except for typescript.
And how's it helping you verify the correctness of your codebase at compile time (it still compiles the source to byte-code before evaluation)? Last time I checked, it didn't care too much about type annotations, and it didn't type-check anything before execution without additional shoehorning.
>No other mainstream whatever
C#, Rust. Notice that I don't even mention my man Haskell or Scala while the latter one is really big.
>literal (dependent) types
Uh, even Java has dependent types, anyway. Read the docs.
>templates
Yes, that's parametric polymorphism. No, it's garbage in C++ to the point where you can't guarantee the soundness of a template. It's done in Java, Kotlin, C#, TypeScript, for the most mainstream ones, not to mention that (and how exactly) it's done in Haskell and Scala.
Edit: How do you feel about type-level programming, anyway? ;)
> And how's it helping you verify the correctness of your codebase at compile time (it still compiles the source to byte-code before evaluation)? Last time I checked, it didn't care too much about type annotations, and it didn't type-check anything before execution without additional shoehorning.
All of the python I write is checked at compile time. Much like java and javac are different tools, python and pytype/mypy are different tools. This is nothing new.
>C#, Rust. Notice that I don't even mention my man Haskell or Scala while the latter one is really big.
Neither supports dependent types.
> Uh, even Java has dependent types, anyway. Read the docs.
No, Java has generic types, not dependent types[0]. Dependent types are types that depend on the value, not the type of an argument. As a concrete example:
Java, rust, and C# don't support this. C++ does for certain types (integers). Python and typescript support this for certain types (integers, strings, enum values).
Other common examples are type-safe matrix multiplication that at compile time can ensure that the matrix sizes all match correctly. Eigen in C++ does this. Haskell and Scala can do this with some massaging, although its not natural. Java, Rust, C# just plainly cannot.
>Much like java and javac are different tools, python and pytype/mypy are different tools
I get where you're heading. The `java` one is for JVM bytecode interpretation and the `javac` is for actual Java. Ok. Cool. I suppose that you do understand it well that leaving the choice on typechecking to external tools is something rather atrocious, do you?
>All of the python _I write_
The factory that produces the hammers for us carpenters provides just hammerheads alone and leaves us to choose and get the grips, so oftentimes fellow woodworkers use them without any grip whatsoever, some make flails and flail the nails, and I just go buy the soft grip from a third-party vendor. What's the problem here?
>[throws a Wikipedia link with the term definition at me, goes on showing me what must be done by a sum type]
You sure do consider me stupid, right? Anyway, what dependent types _usually_ do is they limit a category (think set) to some sub-category. Your example indeed describes dependent types but the problem with it is that it isn't a good place for dependent types. It can be expressed by sum types and I don't see any reason not to do so. By the way, do you have the sum types in Python nowadays?
>has generic types, not dependent types
(1) Look. Have you heard it anywhere that once you get powerful-enough generics, you can at least emulate dependent typing to some extent if not outright get it? Consider this: Java can express Peano numbers at the type level, along with Rust, C# and whatever there is with even remotely working generics. Do you know what that means to us here? Yes. We can express anything number-related at the type level, thus getting one little step closer to dependent typing. The most modest example of what you can do with that is a list with length limited on a type level, which qualifies pretty ok for dependent typing.
>not the _type_
Dependent typing is still about types, and if you work with these on a type level, you will deal with types. The type isn't inferred here but the value is limited by the type in question, so please don't say this, otherwise it will not constitute a sane type system. Imagine the types that just constatate the fact once it happened instead of setting an invariant (ah yes).
>with some massaging
Zero massaging[0] but ok. I hope that, knowing about [0], you've already guessed that there's an implementation of matrix multiplication with actual dependent types[1].
Actually, if you want, I could implement type-level floats for you but I'm not sure if they have any good use case and I'm pretty sure that some operations will compile for a little while: type families operating on numbers are usually defined in terms of recursion. Not sure if we're going to flex type systems at each other at this point, there's a clear winner even if we're not bringing the said to the table: static typing in Python isn't provided out of the box, and while your little example makes me check it out and appeals to me, it's still a third-party tool and that's not how it must be. I will not mention anything about types describing here. If that's not enforced by type system, it's pretty bad but there's no point in discussing that already, enough's been said.
> It can be expressed by sum types and I don't see any reason not to do so. By the way, do you have the sum types in Python nowadays?
I'd suggest you attempt to do so. If you mean that, with the benefit of hindsight, you could go back and replace the string with an enum or set of flags, sure, but no, the signature of the function in question is, and due to historical baggage, must stay
It's just that the set of values `mode` can take on is restricted to a list of ~30 valid strings, which importantly can affect the output type and swap it from Text to bytes. Python already has sum types, and they, uhh, can't solve this problem, as you should know.
> Dependent typing is still about types
Well kind of, in the sense that Literal[4] or Nat::2 is a type, yes. In the sense that Literal[4] is a subclass of int, yes. But it's very different, and much more powerful, because most type systems force you to deal with types as sets of values, and you can't introspect into the sets at all. Dependent types let you introspect and care about particular values, and so you don't have to deal with the types as opaque things. You can care about values, and you aren't restricted to working with only types. But yes the values are still types.
> Have you heard it anywhere that once you get powerful-enough generics, you can at least emulate dependent typing to some extent if not outright get it?
Yes, but implementing peano numerals in java or rust's type system isn't the same as having literal-dependent types, in which you can dispatch based on the actual values present in the language. PEANO_FOUR isn't the same as `4`, and short of a massive amount of codegeneration, you're not gonna have PEANO_FIFTEEN in java generics. Or I mean, if you want to do that, be my guest, but I'll just use `4`.
Or to put it another way: yes, Java's type system is technically turing complete. That doesn't mean that it is pleasant to write programs that execute only during java compilation. Support for literal dependent types makes that more pleasant (and in some cases possible, I'm not actually clear on how you'd write a java function that compiles if passed `4`, and fails to compile otherwise, bridging the gap between PEANO_FOUR and `4` isn't, I don't believe, possible in java. You can no longer use java's own integer class in your code, you have to use your new peano integer class). Native support for dependent types inverts that.
> I suppose that you do understand it well that leaving the choice on typechecking to external tools is something rather atrocious, do you?
This depends. There's a gradient: I run probably 20-30 static analysis tools on my code, from linters to typecheckers to autoformatters. Things that check for malformed TODO comments and all kinds of other fun things. They all prevent various classes of errors. I think not investing in typechecking for large programs is bad, but I also think there's a huge value in gradual typing. Being able to explore interactively in a repl is something that would be painful in java or C++, but is quite pleasant in python. Being able to then convert that code to be production ready by adding types and type-checking, without rewriting in a totally new language, is also quite pleasant. Leaving a user without the choice is also atrocious for many styles of development. That python is able to work for more than one is imo, a feature more than a bug.
No one really wants that in any use cases of type-level numbers. Type-level numbers are generally used to do just opposite: silently (well, loudly, if you ask the compiler to check it) govern the structure in question, limiting, say, its arity or whatever. But if you want to "cross the bridge", I think it's still possible but tricky in case of Java. Don't get me wrong, I don't advocate for Java nor intend to write it, I just use it as a dummy trashcan language to illustrate concepts and their latency in a sense that you don't really need to use fancy X in order to do Y or express Z.
>20-30 static analysis tools
Too bad. I'm usually pretty happy with just plain compiler and a 80-char bar.
>huge value in gradual typing
Doubt that. It's actually a good safety net to stop a project from turning into a big ball of mud but it won't help unless used correctly and with the right intent.
>being able to explore interactively
Hindley-Milner type systems can infer your stuff so well that you usually don't have to worry about supplying type signatures at all. For instance, in Haskell you don't have to "add types" compared to what you do in REPL, you have them all the time and oftentimes you don't have to annotate anything, it's still static and usually done just to give an idea to other people what's doing what.
Oh, not about our little conflict but about Python: does it have some form of traits/typeclasses/whatever? I mean, yes, it has them in form of mixins but can it, in modern days, put a constraint on some type variable, saying that it must have some properties that are described in different traits?
1: No seperator sign like ";", so you cannot express everything you want in one line. Oneliners are often so useful that I don't want to live without them.
2: By default, it caches the bytecode on disk. This is slow and spills those files into the filesystem. I prefer PHP in this regard, which caches the bytecode in memory.
I am currently writing Python based software for a client on serverless architecture. Honestly, I really feel like myself and a lot of people I know use Python only because it has some useful libraries like Pandas compared to other languages. In addition, the effect of 2.xx and 3.xx fiasco can still be felt during development.
These days, I really wish I could write my stuff in Elixir, but too bad its data science eco-system isn't as strong as python's. In the very least, even Ruby as a fallback is much much better to write code with since its philosophy is drastically different from Python's - "Optimize for programmer happiness (Ruby) vs Fuck the programmer and optimze for computers (Python)".[1]
What makes you think it's a cheap troll? Please read the doctrine on how drastically different Python's philosophy is. The interpretation - It's my opinion. Take it or leave it.
Zen of Python. There are good reasons why a lot of the language is the way it is and it optimizes for developer happiness exactly by not letting you do everything under the sun.
Some languages take this to an extreme with its type systems or coding guidelines. For a lot of people Python is a very happy medium for a large subset of a lot of software written out there.
Recently was trying to convert Python to C#... there are few things that are as absolutely incomprehensible as a Python programmer being clever with numpy. I gave up on the conversion. Perhaps this is a point for Python’s flexibility, but I’d say it’s just a waste.
Here’s the code. Try to convert this to any C style language.
mins = X.min(axis=0)
maxs = X.max(axis=0)
idxs = np.random.choice(range(self.dim), self.dim-self.exlevel-1, replace=False) # Pick the indices for which the normal vector elements should be set to zero acccording to the extension level.
self.n = np.random.normal(0,1,self.dim) # A random normal vector picked form a uniform n-sphere. Note that in order to pick uniformly from n-sphere, we need to pick a random normal for each component of this vector.
self.n[idxs] = 0
self.p = np.random.uniform(mins,maxs) # Picking a random intercept point for the hyperplane splitting data.
w = (X-self.p).dot(self.n) < 0 # Criteria that determines if a data point should go to the left or right child node.
return Node(X, self.n, self.p, e,\
left=self.make_tree(X[w],e+1,l),\
right=self.make_tree(X[~w],e+1,l),\
node_type = 'inNode' )
Here is the C# version of that snippet, assuming the existence of such libraries.
var mins = X.Min(axis: 0);
var maxs = X.Max(axis: 0);
var idxs = np.Random.Choice(Enunmeration.Range(this.dim), this.dim-this.exlevel-1, replace: false); // Pick the indices for which the normal vector elements should be set to zero acccording to the extension level.
this.n = np.Random.Normal(0,1, this.dim); // A random normal vector picked form a uniform n-sphere. Note that in order to pick uniformly from n-sphere, we need to pick a random normal for each component of this vector.
this.n[idxs] = 0;
this.p = np.Random.Uniform(mins,maxs); // Picking a random intercept point for the hyperplane splitting data.
var w = (X-this.p).Dot(this.n) < 0; // Criteria that determines if a data point should go to the left or right child node.
return new Node(X, this.n, this.p, e,
left: this.MakeTree(X[w],e+1,l),
right: this.MakeTree(X[~w],e+1,l),
node_type: "inNode")
There is nothing about Python flexibility there other than a possible missing library implementation.
Try converting typical Matlab code to a C-style language and you'll run into similar problems. That's just what you get when your base language isn't all that fast but you have a fast numerics library that allows you to do clever things with arrays and matrices.
Your challenge isn't possible because the code snippet isn't complete. It doesn't include the definition of 'make_tree', 'Node', or the broader context.
That being said, I have no doubt that C++ with Boost would look quite similar to that. The boost libraries have very powerful numeric constructs and can easily handle matrices, uniform random distributions, dot products, and trees.
Well, yeah. My point is the syntax itself is very (too) clever. This syntax is probably possible in C# but you’d be a monster to write it (lots of operator overloads.)
The whole implementation is from an implementation of an extended isolation forest. It’s easy enough to find on GitHub if you’re interested.
The biggest issue, in my opinion, is in dependency management. Python has a horrible dependency management system, from top-to-bottom.
Why do I need to make a "virtual environment" to have separate dependencies, and then source it my shell?
Why do I need to manually add version numbers to a file?
Why isn't there any builtin way to automatically define a lock file (currently, most Python projects just don't even specify indirect dependency versions, many Python developers probably don't even realize this is an issue!!!!!)?
Why can't I parallelize dependency installation?
Why isn't there a builtin way to create a redistributable executable with all my dependencies?
Why do I need to have fresh copies of my dependencies, even if they are the same versions, in each virtual environment?
There is so much chaos, I've seen very few projects that actually have reproducible builds. Most people just cross their fingers and hope dependencies don't change, and they just "deal with" the horrible kludge that is a virtual environment.
We need official support for a modern package management system, from the Python org itself. Third party solutions don't cut it, because they just end up being incompatible with each other.
Example: if the Python interpreter knew just a little bit about dependencies, it could pull in the correct version from a global cache - no need to reinstall the same module over and over again, just use the shared copy. Imagine how many CPU cycles would be saved. No more need for special wrapper tools like "tox".