A few that I'd like to learn about more deeply given the time:
1. Reverse engineering and vulnerability research. Not that much programming in the sense of writing code, but it basically requires you to be deeply familiar with assembly, compiler optimizations, application binary interfaces, and obfuscation methods. It's especially interesting because there are basically two groups that are heavily involved in this work: independent hackers and nation state or adjacent entities (infosec companies, defense contractors, etc).
2. ROM hacking and homebrew. This is similar to reverse engineering, but it's done for more fun reasons. Lots of super interesting work people have done to do things like mod or optimize classic console games, decompile games so they can be recompiled for modern systems, or just learning the development stack of an obsolete console to create new games and software from scratch. It's especially interesting because you tend to have to get into older or more obscure ISAs like MIPS, PPC, or MOS 6502. Check out Kaze Emanuar for some examples.
3. Mainframe computing. It's only esoteric because the hardware and software is largely proprietary and locked down. The Hercules emulator exists though, and IBM lets you run the 30+ year old versions of their Mainframe OSs as a hobbyist without needing to worry about licenses and stuff. The business model sucks, but Mainframes are interesting since they represent a very different paradigm of computing from the now commonplace X86+Linux commodity machines. There's a YouTube channel called moshix which offers pretty accessible long-form information on how to get started hacking around on Mainframe emulators.
Personally, I would say writing compute shaders to do nontrivial tasks like parsing, sorting, and handling data structures. There's a lot of this in Vello. There's a small community of people doing this, which I think will be growing as WebGPU catches on.
Reversible computing. The laws of physics are time reversible but our computers aren't. Quantum computers need to be reversible. Debugging might be easier if we could rewind programs. There are reversible programming languages.
There's also thermodynamics - the more irreversability, the more energy cost. The lower bound approaches zero cost, for computations that jitter forward and backward, where you apply energy to nudge them more often forward.
Quantum computations conserve information. If your gates (transformations) don't conserve information, then you'll break the entanglement and superpositions. At the end of a computation, you do this in a measurement to get an answer.
Audio engineering and signal processing. It’s incredible what you can do. From shaping a waveform that sounds just like a snare drum to geolocating SATCOM jammers by detecting signals below the noise floor of common satellite broadcasts. It’s all just different branches of the same application.
I found FPGA programming a mind-bender, coming from conventional (if low-level) programming.
In sequential software programming, you avoid computation until you need it. With hardware, if you have the logic to do the computation, you use it. Think of having all expressions always evaluated. Instead of conditional evaluation you select which output you want.
This applies somewhat to software on modern processors with abundant functional units (adders, FPUs, load/store units). They're running ahead evaluating as much as possible, then discarding what wasn't selected.
1) Gnu Radio allows you to build signal flow graphs, which are then executed in real time with your hardware. You can even do audio I/O to get an intuition for DSP that just doesn't happen by studying the math.
2) BitGrid is an esoteric computing fabric I've come up with, and wrote an emulator for.[3] It's a grid of 4 bit in and out look up tables, clocked in two phases like the colors of a chessboard.
It's not esoteric to biologists but genetic engineering. Everything is written in assembly and the execution environment is non deterministic and 3 dimensional.
3d computer vision is very cool. You basically take a bunch of pictures of a scene, or a video shot by camera, and do a 3d reconstruction of such scene. The math behind it is tricky. If you manage to code even the basic reconstruction algorithm, you'll grow immensely in math, and math-related coding along the way (assuming no previous background). On the other hand, the job market for such skills is very limited. This is not a good career track, unless one is willing to relocate to Silicon Valley.
So, here's a few pointers of things I came across the past few years, and that you might like:
=== 1: Artifical Chemistries
You could check toward 'Artificial Chemistry',
I bought one book[1][2] out of curiosity a few years ago, and while I haven't really touched it that much, but it's a really interesting perspective on computing that I haven't really seen talked about here (or elsewhere).
Semi-related to the first point is 'Alchemist', "A non-deterministic programming language based on chemical reactions" that was shared here on Hacker News[3] a few years ago.
Still playing with esolangs, I particulary like 'Hexagony' [4], which seems inspired by Befunge[5], but with Hexagons. It has a super cool online IDE [6]. If you don't know about it, then definitely check-out the Esolang Wiki, it's a treasure trove.
Going back to the history of computing, there's the whole soviet side of computing that aren't well known in the west, but probably has fascinating stuff. I only can give you a Stack Exchange question[7] I found a few months ago about some soviet spacecraft, with one answers hinting at some soviet computer languages such as as 'PROL2' and 'DIPOL'.
It's a recurring subject here in HN, but on the off chance that you might be one of today's lucky ten thousand, then check out anything related to APL[8]. It's a fascinating language with extremely powerful construct to play with tabular data. Another APL-related thing is the BQN language[9].
Last one, not my favorite, but definitely high on the 'esoteric' part, is 'Urbit', which is a very ahem special ecosystem that basically eschew anything familiar about the current state of computing. I personally feels that it's weird for weirdness sake, but it's something of a UFO. If you're insterested in Urbit, I would suggest to start with this article titled "Urbit: The good, the bad, and the insane"[11]
1. Reverse engineering and vulnerability research. Not that much programming in the sense of writing code, but it basically requires you to be deeply familiar with assembly, compiler optimizations, application binary interfaces, and obfuscation methods. It's especially interesting because there are basically two groups that are heavily involved in this work: independent hackers and nation state or adjacent entities (infosec companies, defense contractors, etc).
2. ROM hacking and homebrew. This is similar to reverse engineering, but it's done for more fun reasons. Lots of super interesting work people have done to do things like mod or optimize classic console games, decompile games so they can be recompiled for modern systems, or just learning the development stack of an obsolete console to create new games and software from scratch. It's especially interesting because you tend to have to get into older or more obscure ISAs like MIPS, PPC, or MOS 6502. Check out Kaze Emanuar for some examples.
3. Mainframe computing. It's only esoteric because the hardware and software is largely proprietary and locked down. The Hercules emulator exists though, and IBM lets you run the 30+ year old versions of their Mainframe OSs as a hobbyist without needing to worry about licenses and stuff. The business model sucks, but Mainframes are interesting since they represent a very different paradigm of computing from the now commonplace X86+Linux commodity machines. There's a YouTube channel called moshix which offers pretty accessible long-form information on how to get started hacking around on Mainframe emulators.