It was really quite the let down of an article. All that build up, then basically nothing about how adaptive rate limiting would be implemented in practice, how expensive it would be, what it requires, the challenges it involves, etc.
in theory (and also probably in practice) it's a control theory problem and there are already algorithms and ideas out there - here is an article not exactly about rate limiting but implementing something based on this might work really well: https://www.eecs.umich.edu/courses/eecs571/reading/control-t...
Thanks for the feedback - This blog could have been better and less aggressively distributed. At the same time, we should consider the credible body of work behind this. It's not AI-generated spam but backed by actual code that provides a reference implementation behind the idea.
It’s a blog post, it will be judged by its content. It does feel like at a minimum it was heavily edited with an AI assistant, verbose and elegant but light on specifics. And if the goal is to get people hooked on how to implement adaptive rate limiting (and maybe adopt fluxninja) it doesn’t seem to be hitting the right note.
Thanks for the feedback. I agree I could've gone into more technical depth and implementation details. So, taking your feedback in account, I've modified and added more details in the blog. But, I already have a next blog in planning covering the topic in-depth, along the similar lines of your feedback, because of that I didn't intend to make a long one, decided to split the blog :)
No surprise that it's not already implemented because we just throw more cheap hosting and bandwidth at this problem until it's fixed.
The "premature optimization is the root of all evil" misquote really caused a lot of damage to favor growth and profit making instead of better software. It seems programmers have a general habit of just implementing things in the most naive and cheap way and just solve any problem with expensive hardware.
We really need a new trend of fast and minimal software because climate change and a finite planet will not allow us to follow Wirth's law for very long.
The misquote, in my experience, is abusing the "premature" aspect. It's often thrown around as time-based but my understanding of the original quote is that it's all about assumptions rather than data driving complexity in the name of optimization.
Optimization should have measurable improvement and the need for it should be demonstrable. Neither of these is harder than writing a benchmark and/or performing some simple math on data measured for that purpose before going about the optimization.
> "The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming."
and another instance of the same:
> "Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."
I'm not really seeing much of a "misquote" to be honest. I have always understood this quote to mean that spending too much time micro-optimizing things unnecessarily will waste your time and become a burden to you later. Instead, optimizations should only be performed when the need for optimization becomes manifest.
K8S idling will burn much more CPU than any plain OS, still masses of engineers are following the cult of “burn planet, burn“ like in a zombie distopia. Is this a collective brain failure? Bandy Lee should look into destructive tech trends.
I have never really made extensive use of either docker or k8s, but those things always made me feel weird and I always saw those as some kind of shotgun solution to nightmare-ish software who are so complex, badly written and designed that they requires some kind of hazmat suit to work properly.
Not sure it "changed my game" but I have a process that I run on many files at once. In a *nix I would run the processes like so:
for i in list_of_files; do
run_intensive_process &
sleep $(awk '{print $3}' < /proc/loadavg)
done
As more processes start upping my load average, it sleeps longer before running each one in the background. I guess GNU parallel can do this sort of thing with a load-sensitive command line arg, but I find this pattern easy to remember, and parallel's decidedly not.
at(1) runs jobs after a specified time. (A common mistake is to presume that "at" means "will run when specified" when in fact it means "will run no earlier than specified", and depending on system characteristics, could run substantially later.)
batch runs jobs ... as system load permits, without any time specification, in a FIFO queue.
There's also cron, the third Unix scheduler, which executes a set of specified tasks on a regular schedule. All three are related.
I have the idea that at is more widespread than batch, and since you can say 'at now' (or 'at $(date)'), at is a better bet.
As far as I know, cron doesn't pay attention to load average, but I would love to be corrected.
All of this feels like a very crufty and highly neglected part of Unix, which is a shame. There's an alternate history of Unix where it's all about batch jobs and datagram Unix domain sockets.
at and batch are both part of the same underlying facility. The former seems more known than batch, but where you find one you will pretty much always find the other.
On many systems they'll share an integrated manpage.
It has to be behavioral, not IP based for the most part because of the ways these networks operate—primarily on personal computers/routers on US-based residential connections.
I deal with these constantly for our clients. It’s amazing how smart they function to get around limits (thousands of IPs and valid recent user agents) but how they miss easy things like using the same cookie to submit 1000s of different entries with obvious purchased lists of names/emails, many of whom are deceased.
"Thousands of IPs" can not hurt Twitters bottom line in any way. With a "read 10 tweets per IP per day" limit, that is 10k tweets. Delivering 10k tweets does not even cost a penny.
https://brightdata.com/proxy-types/residential-proxies gives you access to 70 million IPs. You'd have to be a little insane to scrape millions of tweets like this though because the bandwidth costs would be very high as even with massive volume you're probably not getting any price lower than $5-7/GB. Fortunately Twitter doesn't rate limit by IP (at least as long as you're not posting), they allow guest access and they have an endpoint that lets you get 100 tweets at a time so it's not very difficult to scrape as I described in https://news.ycombinator.com/item?id=36667225.
10 per day is a ridiculously low limit. Also, NAT and shared networks (think college campus) mean that hundreds of people often share the same IP, making that limit even more ridiculous.
Reality would be that limit would have to be much higher.
Is it common for mobile networks to hand out short (64 or smaller) prefixes to phones? Phones are routers after all (hotspot) so this would seem to break stuff.
My solution is to make it less of a walled garden. At the moment, they often refuse even reading a single tweet when not logged in.
And for the "simply log-in": It is even more expensive (way more expensive) to get good email adresses as it is to get good IPs. Email providers have very tough checks these days before they give you one.
The biggest problem with adaptive rate limiting is that, in many contexts, clients value predictability more than they do peak performance. Adaptive rate limits change client behavior in ways that clients can't predict or control.
Even in networking, the classic application domain of adaptive rate limits (like TCP's algorithms), core networks are built to offer predictable performance. Edge networks, like public WiFi, that offers best-effort performance with adaptive rate limits, are often met with frustration and even anger the moment those limits kick in.
I do not know why, but by default the body element has an overflow of hidden set via css. There is probably a good reason to do this that I do not know about. Would love to know the reason behind that from a WebDev as it strikes me as unnecessary having seen the page with JS and without JS and not being able to see much of a difference.
