> FP will always occupy a niche because of where it sits in the abstraction hierarchy
At some point in history, people stopped worrying about not understanding compilers, how they allocate registers and handle loops and do low-level optimizations. The compilers (and languages like C or C++) became good enough (or even better than humans in many cases) in optimizing code.
The same happened with managed memory and databases, and it will happen here, too. Compilers with FP will become good enough in translating to the machine code so that almost nobody will really care that much.
The overall historical trend of programming is more/better abstractions for humans and better automated tools to translate these abstractions into performant code.
Some people stopped worrying about not understanding compilers. They're not working on drivers, realtime (esp where low lag & jitter are concerned such as motion control), or high performance software of all stripes, trying to squeeze the most out of the available hardware. It's all about choosing the right tool for the job, and there is no right tool for every job. A guy generating sales reports has very, very different needs from the lady writing a rendering engine.
Michael Abrash (graphics programmer extraordinaire) said it best, and I'll paraphrase: the best optimizing compiler is between your ears. The right algorithm beats the pants off the most optimized wrong algorithm. Or, as i like to say "there is nothing faster than nothing" -- finding a way to avoid a computation is the ultimate optimization.
And managed memory is wonderful, almost all the time. That is, just until the GC decides to do a big disposal and compaction right in the middle of a time-sensitive loop causing that thing that "always works" to break, unpredictably, due to a trigger based on memory pressure. Been there, done that. If it's a business report or a ETL, big deal. If it's a motor-control loop running equipment, your data or machinery is now trash.
For most of the programming world, and I count myself in this group, the highly abstracted stuff is great. Right up until the moment where something unexpected doesn't work then it turns in to a cargo cult performance because it's nearly all black-box below. Turtles, all the way down.
There is value in understanding the whole stack, even today.
> Some people stopped worrying about not understanding compilers.
But no, it's not some people, it's not most people, it's 99%+ of all developers that stopped worrying about compilers. There will always be a use case for it, but when we're talking about < 1% of all developers we're really spending time talking about a niche.
There will always be niches in any industry, but we shouldn't design our industry/profession about niche cases.
While I 100% agree with you, as a Java dev who is very interested in optimization (particularly register coloring) at a certain point I think you have to realize that any "compiler type" optimizations you do (Ooh, I'll optimize my variable declaration order to not spill to memory!) is just ignored and re optimized by most compilers worth their weight in salt. Therefore, it's totally counter productive. All the time spent worrying about GC lag is, IMO, wasted compared to other more productive things. I haven't programmed anything mechanical in over 6 years. Basically, for your average developer, while I highly recommend learning the whole stack, I don't believe the notion that understanding the whole stack will actually lead to tangible improvements for programmers. They'd be better served focusing purely on theory (and by theory I mean algorithms).
As a side note: I hate hardware, but I love graph algorithms, which is why I love register coloring so much :)
well as always when discussing functional programming people are not discussing the merits of functional programming but rather the merits of basically throwing away objective programming in favor of functional programming.
That is ofcourse complete nonsense but modern computer science politics like all modern politics only deals in absolutes
Programming is has grown so much as a field that generalizations like this rarely capture the truth.
It's true that in many domains, people care much less about performance than they used to.
At the same time, other people care a lot more about performance. Programming is just big and diverse.
The end of single score scaling is one big reason it's more important than ever.
Another reason is simply that a lot more people use computers now, and supporting them takes a lot of server resources. In the 90's there were maybe 10M or 100M people using a web site. Now there are 100M or 1B people using it.
I think there's (rightly) a resurgence in "performance culture" just because of these two things and others. CppCon is a good conference to watch on YouTube if you want to see what people who care about performance are thinking about.
----
If you're writing a web app, you might not think that much about performance, or to be fair it's not in your company's economics to encourage you to think about it.
But look at the hoops that browser engineers jump through to make that possible! They're using techniques that weren't deployed 10 years ago, let alone 20 or 30 years ago.
Somebody has to do all of that work. That's what I mean by computing being more diverse -- the "spread" is wider.
The point here is that for many domains performance is not the top consideration. And it's also worth noting that it's perfectly possible to tune applications written in FP languages to get very good performance. It's also possible to identify the parts of the code that are performance critical and implement those using imperative style. This is especially easy to do with Clojure where you have direct access to Java.
So, yeah if you're working in a niche domain where raw performance is the dominant concern, then you should absolutely use a language that optimizes for that. However, in a general case using FP language will work just fine.
Sure I get that, but I'm saying your statement about "the overall historical trend" is wrong, or at least fails to capture a large part of the truth.
At some point in history, people stopped worrying about not understanding compilers
This part is misleading too -- I would say there is a renaissance in compiler technology now. For the first 10 years of my career I heard little about compilers, but in the last 10, JS Engines like v8 and Spidermonkey, and AOT compiler tech like LLVM and MLIR have changed that.
The overall historical trend is that computing is getting used a lot more. So you have growth on both ends: more people using high level languages, and more people caring about performance.
It's not either/or -- that's a misleading way of thinking about it. The trend is more "spread", not everyone going "up" the stack. There will always be more people up the stack because lower layers inherently provide leverage, but that doesn't mean the lower layers don't exist or aren't relevant.
And lots of people migrate "down" the stack during their careers -- generally people who are trying to build something novel "up stack" and can't do it with what exists "down there".
Once parallel execution became part of every design discussion that had a performance concern, the vast majority of programmers stopped caring (and consequently talking) about compilers. Who cares if you can do 36k requests/s to my 18k if I can do 36k across 2 or 3 machines? I pass that on to the customer. Why try to hire for or wait around for an optimization trick to double performance (that will likely never be realized or discovered) when there's business to be done? The post-optimization is pure profit and can be quantified, so might as well wait and let some specialist (who does care about compilers) handle it if the product ends up being valuable enough to hire them. This is how development and hiring works today.
> At some point in history, people stopped worrying about not understanding compilers
> This part is misleading too
Not in the least. Interpreting that to mean "all people stopped worrying" is deliberate misinterpretation.
You're simply used to working in environments where technical excellence doesn't exist. In such environments performance is not a big concern, but neither is quality in general...
Over-indexing on performance has nothing to do with technical excellence. It having the the wrong priorities.
Saying people who don't optimize for performance don't have technical excellence is just like saying people who don't get all of their program to fit into 32kb don't have technical excellence.
Yes it requires skills to get a program to run in such a small amount of space, just like it takes skill to perform detailed performance optimizations. But in either case if that's not your job you're wasting time and someone else's time, even if it makes you happy to do so.
A product is designed to serve a purpose, if instead of working on that purpose is developer is squeezing out a few additional cycles of perf or a few additional kb of memory they have the wrong priorities.
No that doesn't mean go to the other extreme, but choosing to not spending unnecessary time on performance or size optimization is entirely unrelated to technical excellence. And any senior engineer knows this.
> At some point in history, people stopped worrying
Did they ? Because I keep seeing people around me who want to get into FPGA programming because they aren't getting enough juice from their computers. Sure, if you're making $businessapp you don't care but there is a large segment of people who really really really really want things to go faster, with no limits - content creation, games, finance... hell, I'd sell a rib for a compiler that goes a few times faster :-)
The point is that to be mainstream it's enough to be used by one major app store
How many apps care about FPGA or what compilers do, given that they don't even know what the underlying OS does or when and why memory is allocated?
I work in finance, even there performances are for niche projects, the bulk of the job is replacing excel macros with something slightly less 90s style.
Fun fact: C used to be considered a high-level language. Now everyone talks about it being "close to metal" which to olds like me is a dead give-away the person either doesn't know C or doesn't know the "metal". Most of the stuff people think of as being the "metal" in C are, in many cases, virtual abstractions created by the operating system. Embedded development w/o dynamic memory allocation less so... but that's not what most people are talking about.
Well it depends on what side of the kernel/userspace boundary you are talking about doesn't it.
While C for userland programs may need to conform to the operating system's libc and system call interface abstractions, on the other side of the syscall boundary is C code (ie. the kernel) that is indeed very "close to the metal".
Running any code in the CPU's most privileged ring, regardless of language, is going to give you access to directly programming the MMU, scheduling processes across multiple CPU's, control over caches to a fairly large extent, and the ability to control every device in the system. A large amount of this is achieved by bit-banging otherwise inaccessible registers in the CPU itself via assembly (ie. in the case of the MMU, GDT and IDT for x86), or via MMIO for modern busses/devices. The language doesn't necessarily "need" to have a complex model of the entire physical machine to be able to achieve all of those things. How much closer to the metal do you want to be?
You really want your programming language to have innate constructs for directly controlling the baggage the x86 CPU (or any other for that matter) brings with it? I don't.
You also want kernel code to be performant (ie. compiled by a decently optimizing compiler, of which there are many for C), allow you to disable garbage collection or be totally free of it so you can explicitly manage separate pools of memory. C ticks all those boxes which is why its still the most dominant and widespread language for OS kernel development nearly half a century since UNIX was first rewritten in C, and will be for years to come, like it or loathe it, and despite there being much more modern contenders (eg. Rust) which don't have the momentum yet.
All of that is given access by Assembly opcodes, not C specific language features.
And if you going to refer to language extensions to ISO C for writing inline Assembly, or compiler intrinsics, well the first OS written only in high level language with compiler intrinsics was done 10 years before C existed and is still being sold by Unisys.
The only thing that C has going for it are religious embedded devs that won't touch anything else other than C89 (yep not even C99), or FOSS UNIX clones.
And yeah, thanks to those folks, the Linux Kernel Security summit will have plenty of material for future conferences.
Which modern, portable language gives you direct control over the MMU, out of order execution, delay slots and explicit cache access, other than hardware specific assembler? None that I know of can do this in a hardware agnostic way. Do tell.
I clearly mentioned that assembler was required for much of this, where components aren't programmed by MMIO. This would be the same regardless of whether you used Rust, Go, or FORTRAN77 to write your kernel.
I'm not even going to bother with your security comments, we all agree by now. There are plenty of people using C99 in embedded at least in userspace, even the Linux kernel uses some C99 extensions (eg. --std=gnu89 with gcc), and those FOSS UNIX clones have taken over the planet at this point in terms of smartphone adoption, data center servers etc. Despite the obvious flaws, this is still a better world to live in than Microsoft's proposed monoculture of the 1990's.
None, including C, which makes it nothing special. Any compiled language can call into Assembly.
The phones I know as having taken over the world run on C++, Objective-C, Swift, Java, with very little C still left around, and with its area being reduced with each OS release.
As for data centers, there is a certain irony that on Azure those FOSS run on top of Hyper-V, written in C++, on Google Cloud run on top of gVisor written in Go, on Amazon on top of Firecracker written in Rust, and on ChromeOS containers written in a mix of Go/Rust.
Hyper-V runs as a Windows Server role. Windows kernel is C, and assembly
gVisor runs on Linux -> C, assembly
Firecracker runs on KVM, which runs on Linux -> C, assembly
In every single thing you have listed, the closest thing to the "bare metal" is C, and assembly. THAT's what makes C special. Its level of adoption, ubiquity and accessibility. Not its spectacular lack of security risks.
Anyway, you have come a very long way from where the parent poster started which was:
Most of the stuff people think of as being the "metal" in
C are, in many cases, virtual abstractions created by the
operating system.
To which I merely pointed out, on the other side of the interface layer is, most commonly C. And assembly.
Operating Systems design has to and is obviously evolving away from this. I disagree that we have reached "peak C" and that is going to decline before it gets bigger.
Unfortunately pjmlp many of the conversations we have start this way, and devolve into this. I don't think I'm going to bother again. I think one (or both) of us will just have to agree to disagree. Have a nice day.
> None, including C, which makes it nothing special. Any compiled language can call into Assembly.
I wish you joy and entertainment interfacing your managed data structures with assembly code.
Yesterday I was forced to look into COM for the first time. There was some kind of callback that I was interested in, and it had basically two arrays as arguments, only in a super abstract from. I'm not lying, it was 30 lines of code before the function could actually access the elements in the arrays (with special "safe" function calls to get/set data).
Of course, that stupid callback had to be wrapped as a method in a class, and had to be statically declared as a callback with a special macro that does member pointer hackery, and that has to be wrapped in some more BEGIN_MAP/END_MAP (or so) macros. Oh yeah, and don't forget to list these declarations in the right order.
Thanks, but that's not why I wanted to become a programmer.
Not that I want to vehemently disagree with your security statements, but I think I'd love to have a little bit more "show" and less "tell". That also applies to showing practicality of managed languages, practicality of 90's business software development (C++/COM), practicality of dead commercial languages (Delphi + VCL).
Giving just endless lists of ancient buzzwords doesn't help.
> It is coming for sure, I have not forgoten about it, I just have a private life to take care of, you know?
I was thinking you'd look at it before writing your next 25 comments, but it seems I was wrong. So I'll just wait, it's fine.
> The 21st century React Native for Windows is written on top of COM/C++
From a skim I could find exactly zero mentions of COM/C++ stuff in there. Sure, this RN might sit on a pile of stuff that has COM buried underneath. That doesn't mean that COM is a necessity to do this React stuff, and not even that it's a good design from a developer's perspective.
You give zero ideas what's a good idea about COM. Just buzzwords and links to stuff and more stuff, with no relation obvious to me.
If you actually have to go through the whole COM boilerplate and the abominations to build a project with COM, just to connect to a service, because some people thought it wasn't necessary to provide a simple API (connect()/disconnect()/read_next_event()) then the whole thing isn't so funny anymore.
Maybe I was unclear, but it was a C++ program (dealing with macros, as I said - VARIANTS and DISP_IDs and PROPERTIES and stuff). No joy to use.
As for other languages, I haven't touched COM at all but the idea of making GUIDs for stuff and registering components in the operating system doesn't seem a good default approach to me. Pretty sure it's more reliable to link object files together by default, so you can control and change what you get without the bureaucracy of versioning, etc.
> ReactNative for Windows uses WinUI and XAML Islands, which is UWP, aka COM.
Is the fact that COM is buried under this pile more than an unfortunate implementation detail?
Some engineers have terminal systems brain. That's a rude way for me to say it, but I have met engineers who feel the need to fully understand how code maps to hardware otherwise they don't feel comfortable.
At some point in history, people stopped worrying about not understanding compilers, how they allocate registers and handle loops and do low-level optimizations. The compilers (and languages like C or C++) became good enough (or even better than humans in many cases) in optimizing code.
The same happened with managed memory and databases, and it will happen here, too. Compilers with FP will become good enough in translating to the machine code so that almost nobody will really care that much.
The overall historical trend of programming is more/better abstractions for humans and better automated tools to translate these abstractions into performant code.