501 is "not implemented", I don't see how that's relevant? The server sends a "401 Unauthorized", which is what prompts the browser to show the password prompt. The thing is, I've seldom seen such a page -- because the browser shows me a login prompt.
On face value, I tend to agree with the OP that it would be cleaner if we could use HTTP auth for real things. In practise, there's simply so much missing from the equation, combined with a very mature selection of software that makes this very easy to do in any webapp, to be something worth wasting efforts on.
Is it really even cleaner? I understand why people say that --- because web services clients usually don't have a notion of "cookies" --- but at pretty much the same level of security and convenience as HTTP Authentication, you can also issue persistent access tokens. I don't see how a token is in any way inferior to embedding authentication in an HTTP header.
It's cleaner because you, in theory, can outsource an entire functionality class to your infrastructure. Less complexity for you to deal with.
A bit like serving static files - you put your files in a directory, and they're not only served, they're served faster, better and more reliably and scalable than whatever you can code as part of your app.
I don't understand the end to end argument then? I'd say it's a huge win for cleanliness that I never need to touch a buffer in my life, but can still build and deploy high performance web apps, because I can build on the Apaches/nginx/whatevers of the world.
Requiring users to log in in modern web stacks is indeed just a question of setting a flag, and it's done for you, you never need to know about, let alone consider encryption of, session cookies, so this is outsourced down the stack (and sufficiently clean for anything I can think of). Just not as far down as the HTTP standard suggests is possible.
On face value, I tend to agree with the OP that it would be cleaner if we could use HTTP auth for real things. In practise, there's simply so much missing from the equation, combined with a very mature selection of software that makes this very easy to do in any webapp, to be something worth wasting efforts on.