Having a job means that you have to do two main tasks. The first is doing whatever your job description says. If you're an engineer, you have to build widgets; if you're in sales, you have to sell widgets; if you're a designer, you have to design better widgets. The second is doing whatever your manager says. Unfortunately, it's not rare that your manager makes your job worse. If you're an engineer, your manager might tell you to build subpar widgets. You could do what your job description says and build good widgets instead, but that won’t get you promoted.
The Neuralink compression challenge (https://content.neuralink.com/compression-challenge/README.html) is a good example of this. The challenge asks members of the public to design a data compression algorithm that compresses data by 200x. In other words, the algorithm should take 200 pieces of data and then compress them down to 1, with the ability to re-extract all original 200 pieces of data.
The challenge is impossible.
Most data compression algorithms can compress data by 2x to 5x. This includes zip, which we all know and love. The state-of-the-art is only slightly better: the current winner of the Hutter Prize (http://prize.hutter1.net/), a nearly two decade old prize to encourage the development of better compressors, achieved compression by 9x. 200x is a pipe dream.
What’s worse is that the challenge doesn’t merely want entrants to design an amazing compression algorithm. Remember, the challenge is being run by Neuralink, a company that puts implants in brains. The algorithm will be used to compress signals coming out of the brain. Because the brain works fast, the signals have to be compressed extremely fast, specifically, at the rate of 200,000 bytes every millisecond. The state-of-the-art Hutter Prize winner from up above would take 36 times as long to compress that data.
What’s even worse is that, obviously, you can’t put a big battery inside a brain. The risk of a big battery overheating or leaking is too high. So the brain implant runs on very little power, about one-thousandth of what an LED light bulb uses. The compression algorithm has to run on this very small, very weak implant. Keep in mind, the compression algorithm can only use a fraction of the power the implant is running on, because the entire contraption also includes other pieces like a radio for actually transmitting the compressed brain signals.
In other words, the challenge wants entrants to not only do something impossible, but also do it quickly and efficiently.
For now, let's pretend your boss gave you this problem. Don't worry about questions like “Why am I solving this impossibly hard problem for free?” and “If I figure out how to do this, why wouldn't I start my own company and become a billionaire?”
What should you tell your boss? If you're being honest, you would say something like “this won't work; we need to start over.”
A Twitter user working on the problem did exactly that. With somewhat inartful phrasing, he said (https://x.com/lookoutitsbbear/status/1794962035714785570) that he came up with a solution that simplified the brain signal before compressing it. That is, instead of compressing all 200 pieces of data, his algorithm compresses fewer. Does this fulfill the terms of the challenge? No. Hoards of Twitter users descended upon him to tell him this over and over. The challenge doesn't allow simplifying the brain signal.
However, in the real world, this is absolutely the kind of cludge a good engineer would implement. Instead of building a machine that collects 200 pieces of data but can't do anything with it, a good engineer should build a machine that collects less data and transmits it so it can be useful. @lookoutitsbbear’s solution will lose the challenge, but something like it will be essential for getting Neuralink working.
Except, you know, this information won't make the boss (or the Neuralink challenge or its fans) happy.
If I tell you to write an image compression algorithm, you aren't going to be able to do much with a bitmap of uniform randomly generated pixels. However if I tell you that in the domain I'm working in there are only two colors white and black, immediately I can reduce storing each pixel from 24 bits to 1bit, saving a factor of 24. If I tell you further that >99% of pixels are going to be black, more compression tricks become possible, etc.
I don't have expertise in this particular problem, but a priori dismissing it by comparing to Hutter is not valid.