In summary: someone with a GitHub account discovered that you can hijack the C pre-processor (give or take some GNU language extensions) to transpile BASIC/PASCAL/ALGOL like languages to C. They call their macro collection "evil" and ramble about destroying C, trying to create some form of mystique I guess.
The idea by itself is nothing new and pretty much any novice C programmer eventually stumbles across it. Famously, the original Bourne Shell was written in a variant of ALGOL68, fed through a C compiler using macro based translation[1][2], which helped spark the idea for the IOCCC[2].
>They call their macro collection "evil" and ramble about destroying C, trying to create some form of mystique I guess.
It read to me more like a tongue in cheek statement in the same way that a car enthusiast might joke that they're about to destroy a Honda Civic by putting in a V8 engine and installing a body kit.
If there were a novelty requirement to push to github (or to publish on HN for that matter) it would be much less fun. On the other hand, it is also fun to hear about the history.
Personally I always enjoy abusing the C preprocessor for fun and profit. Just yesterday I was contemplating the potential amusement of using cpp (and cc) to generate custom assemblers, customized assembly language syntax, and assembly language interpreters.
Yeah, it was just some fun. It's 'evil' because it should never be used for business. And yes, Bournegol was an inspiration and I mention it in a few places on the gist.
I did also take it too far and wrapped it with tcc [0] into a "language".
Well, there already are other, operational space planes besides the X-37B. Namely the Shenlong[1] (3 success full missions) and the CSSHQ[2], the later being currently in orbit (3rd mission, launched in December 2023).
But we don't know that much about those either, as they are kept in just as much secrecy as the X-37B.
I'm a bit surprised the bulk of the comments here seem to fixate onto Go specifically, seemingly missing the actual point of the article:
The article demonstrates the Usermode Driver API, showing how a Linux driver can offload work into userspace (with or without access to a working filesystem).
I mentioned this yesterday in the context of the in-kernel codec discussion[1], where the questions "can't this be done in userspace" or "why not sandboxing" were dismissed pretty quickly.
Their rationale for Go is explained in the article.
> Strictly speaking, any program will do, but we need to ensure that the program in question has no dependencies on the file system. Linking it statically provides benefits. Go programs are statically linked by default, and to illustrate that the following approach works with any kind of program, we have chosen to embed a Go program into the kernel.
There were also some Java CPUs, which directly executed bytecode, in which case a "Java driver" would just be the lowest, system-level language available: https://en.m.wikipedia.org/wiki/Java_processor
The kernel has API in place to run code in userspace from within a kernel module ("usermode driver"). That would in theory be one way something complex in the kernel could be isolated. The obvious downside being the additional context switches and round trips for the data adding latency.
Here's a blog post that demonstrates embedding an entire Go program as a blob into a kernel module and running it in userspace from the module:
While this is definitely an interesting idea (if with a lot of sharp edges, some of which you’ve discussed) some of these things are difficult to actually sandbox like this. Like, how do you meaningfully sandbox something that sets up page tables? Yes, it’s running in userspace–but if I exploit it there then I have extremely access to the system, largely equivalent to it running with kernel privileges in the first place. Likewise code that manages where the processor comes out of reset effectively has privileges regardless of where it runs, because the processor comes out of reset in a privileged state and influences on its execution are largely equivalent to having that privilege.
It is utterly unrelated to the ongoing issue (which I'm 100% sure is the reason this is on the front page for a second time within 12 hours right now).
The author is also the developer of lzip, while it does have some valid points, one should apply a necessary level of skepticism ("Competitors product is really bad and unfit", says vendor).
While you are saying that it is utterly unrelated, I'll respond with "that not only related, this is a root cause of current situation".
We have a single massively used implementation (also known as reference), which does not even follow xz format specification. It is autohell-based, which allowed m4 injection. In other places it is CMake-based, but uses autohell-based code, inserting hidden dots where they could just use https://cmake.org/cmake/help/latest/module/CheckSymbolExists...
"Poor design" is still there, as a core of library. It is as cool, 64-bit square roots and 128-bit divides in Bcachefs[1]
Spoiler: this lbzip2 code produces corrupted files in some cases, should we care if it is a backdoor? Or as usual, disable optimizations, disable valgrind, disable fuzzers and say that everything is ok?
I think that's probably it. I.e. it's a nobody-cares issue. Looking at the older posts, there's some serious issues with lzip too (looks like corruption-related) that got a less than stellar response from the author.
Just quickly tested lzip vs zstd -9 on a 100 megabyte text file. zstd is almost as good but many times faster. I wonder if the lzip author's work got obsolete.
I skimmed the manual of lzip a bit. The author likes to talk a lot about how everything is done correctly in lzip and there's this line line "The lzip format specification has been reviewed carefully and is believed to be free from design errors. " If you type lzip --help it talks about how it's better than bzip2 or gzip.
Maybe they are right but ugh comes off real arrogant.
1. both lzip and xz are using lzma compression library internally, so there is no difference in their compression ratio/speed
2. lzma compression is LZ + markov chains, while zstd is LZ + order-0 entropy coder (similar to zlib, rar and many other popular algorithms)
markov chains are higher-order entropy coding, i.e. one using context of previous data. it's slower, but sometimes gives better compression. but text files don't get any benefit from it. OTOH, various binary formats, like executables or databases, get significantly better compression ratio. in my tests lzma-compressed binary files are ~~10% smaller on average.
so, many claims that zstd and lzma provides the same compression ratio, are based on testing on text or other files that don't benefit from higher-order entropy coding. of course, I imply maximum-compression setting and equal dictionary size, in order to make fair comparison of compression formats rather than particular implementations.
(I'm author of freearc and several LZ-based compressors, so more or less expert in this area :)
Because xz archives the best compression ratios. If you care about archive size above all else it's the best choice. If you use "medium levels" I agree there's no point, zstd is superior in that regime (achieving faster compression and decompression for these compression ratios)
Any compressed format is okay as long as you have a working compressor and decompressor, where the compressor can reliably compress an input and the decompressor can reliably decompress the compressed input. Everything else is not as relevant, including the exact file format (because you have the known-good decompressor).
xz is not worth using at medium levels. At its highest levels, it sometimes just barely has an edge over the highest levels of zstd, and it existed before zstd did.
Surprisingly, the article doesn't seem to mention SquashFS[1] or EROFS[2].
Both SquashFS and EROFS are filesystem specifically designed for this kind of embedded, read-only use case. The former is optimized for high data density and compression, and already well established. The later is comparatively new and optimized for high read speed.[3] SquashFS as a rootfs can already be found in many embedded applications using flash storage and is typically also combined with tmpfs and persistent storage mount points or overlay mounts.
For both those filesystems, one would build a rootfs image offline. In the Debian ecosystem, there already exists a tool that can bootstrap a Debian image into SquashFS[4].
This is actually why I switched all my Pis to Alpine Linux since they use a RAM disk by default for RaspPis Haven't lost an SD card since, meanwhile my Octoprint RasPi has failed 4 times (mostly because of random power disconnects)
I'm a fan of Alpine too, but I want to point out that to the best of my knowledge at least, running root from RAM is trivial and is not something you need to change distros for, you simply have to add the "toram" kernel boot parameter. For example, if your entire read-only root is contained in "filesystem.squashfs", then adding "toram=filesystem.squashfs" would read all of this into RAM on boot.
I can't vouch for every single distribution out there, but I've done this successfully on Debian at least.
Very true. Before I found Alpine I used a heavily modified raspbian with a root read-only ramdisk which also worked but Alpine did it out of the box without having to modify everything
> ... can change during the lifetime of a process, ...
Fun fact: this is already an issue for the vdso.
gettimeofday(2) can use the CPU builtin cycle counters, but it needs information from the kernel to convert cycles to an actual timestamp (start and scale). This information too can change during the runtime.
To not have userspace trampled over by the kernel, the vdso contains an open-coded spinlock that is used when accessing this information.
I learnt about this while debugging a fun issue with a real-time co-kernel where the userspace thread occasionally ended up deadlocking inside gettimeofday and triggering the watchdog :-)
Does avoiding userspace being trampled by the kernel mean avoiding a context switch? Does the kernel write to userspace-accessible memory that the spinlock guards?
Unrelated, but what does "open-coded" mean? I never seem to find an obvious answer online.
I have a copy of "Radiosity, A programmers perspective" sitting on my shelf and can recommend it. IMO it does a good job explaining the fundamentals and concepts that Ian Ashdown himself is quoted in the article to have struggled with initially.
It is one of those books, where the author step-by-step pieces together a program and has the explanations structured around that. In this case, it's a radiosity renderer written in C++ called "Helios". At the time of writing, 32 bit Windows was the hot new shit, so the book takes a few detours explaining Windows API and along the way also develops a tiny software rasterizer for visualization.
Sadly, I no longer have the floppy that came with the book and the website (helios32.com) seems to be defunct. It's indexed in the wayback machine tough. The book appears to have been freely available through website for some time already, but I cannot find a snapshot of the source code.
What happened to radiosity ? I remember it being all the rage in the late 90s before other techniques took the market. Was it a performance matter (maybe radiosity being too complex/cpu intensive) or was it a physical model limitation (you can't simulate all light phenomenons) ?
Basically the idea behind radiosity is, you divide the surfaces in your scene into discrete patches (i.e. tiny rectangles mapped onto everything). A patch receives energy from a light source, or reflected energy from other patches, and it reflects all of that back into the scene (conservation of energy).
The assumption is, that a patch is perfectly diffuse and it reflects in all directions equally. Imagine a fish-eye-lens perspective (actually a hemisphere) from a patch into the world. The projected size and visibility of all the other patches gives you the proportion of energy transferred to/from that other patch.
> Was it a performance matter ...
Computing the final color of a patch is somewhat computationally expensive. It involves either inverting a giant matrix, or iterating until you reach a stable result. The end result is not a rendered image, it's a perspective independent light distribution in your scene. You still need to combine that with another rendering technique, like ray-tracing or rasterization to actually get a picture.
Computer games of the era you mention (e.g. the Half-Life series and other GoldSrc/Source engine based games), did radiosity in an offline, pre-computation step and stored the patch colors in a texture (this is usually called a light map). Using the light map at runtime is pretty cheap.
> ... or was it a physical model limitation (you can't simulate all light phenomenons) ?
As I described above, radiosity assumes perfectly diffuse reflectors. It can't do mirror-like specular reflection of light. Ray-tracing and radiosity actually complement each other really nicely in that aspect, as mirror like reflections are trivial to do with ray-tracing and you can integrate light maps for indirect, diffuse bounces.
That said, with modern hardware, doing radiosity in real-time even for reasonably sized scenes has become feasible. There is e.g. the proprietary "Enlighten" engine that claims to do real-time radiosity. There are other approaches that have become feasible as well, like doing real-time photon mapping. Ray tracing hardware finally being a thing is also certainly a game changer. Well over a decade ago, Crytek did some work on voxel grid diffusion based stuff and Nvidia worked on integrating cone-tracing into that. I have little idea tough what modern game rendering pipelines look like, I've been pretty much out of touch with that industry for quite a while now.
Also radiosity for offline rendering got largely replaced by sampling based techniques, first things like photon mapping and more recently by path tracing. Sampling approaches are easier to parallelize and generalize better to other types of materials.
the perpsective independent aspect fascinates me, especially in the era of massive multi party applications, where it seems a lot of subspaces are recomputed redundantly. anyway just a brain fart, thanks a lot for the answer
Radiosity was mostly used by laypeople (non-rendering specialists) to mean global illumination in general.
The actual specific radiosity algorithm (diffuse only, form factors and matrix solving etc) proved to be too unwieldy, requiring huge amounts of effort and care to get good results. This is in contrast to Monte Carlo path tracing, which doesn't require extreme care with meshing, doesn't assume diffuse BRDFs and is really well suited to modern massively parallel architectures. Consequently it completely took over production rendering, and game rendering is next.
You need to dice your scene in many well formed elements (it's tricky, lots of edge cases), and you are simulating diffuse light only. Point sampling achieve the same results without a complex preprocessing step.
To get good results on a complex scene you need to increase the subdivision count of the radiosity calculation by a lot, which will be slow unless you're smart about it.
And radiosity only accounts for diffuse bounces, leaving out glossy and specular reflections.
It's still used when you need accurate energy transfer. Plenty in real lighting software (Dialux, Agi32). But there are faster methods if you just want it to look good.
The idea by itself is nothing new and pretty much any novice C programmer eventually stumbles across it. Famously, the original Bourne Shell was written in a variant of ALGOL68, fed through a C compiler using macro based translation[1][2], which helped spark the idea for the IOCCC[2].
[1] https://en.wikipedia.org/wiki/Bourne_shell#History
[2] https://en.wikipedia.org/wiki/International_Obfuscated_C_Cod...
reply