Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range, whereas in the ion channels of a synapse it is in the picoampere range. The voltage differs by roughly a factor of ten. So the wattage differs by a factor of 10^10.
One important reason for the difference in current is that transistors need to reliably switch between two reliably distinguishable states, which requires a comparatively high current, whereas synapses are very analog in nature. It may not be possible to reach the brain’s efficiency with deterministic binary logic.
>the current in a CPU transistor is in the milliampere range
? you sure about that? in a single transistor? over what time period, more than nanoseconds? milliamps is huge, and there are millions of transistors on a single chip these days, and with voltage drops of ... 3V? .7V? you're talking major power. FETs should be operating on field more than flow, though there is some capacitive charge/discharge.
Single transistors in modern processes switch currents orders of magnitude lower than milliamps. More like micro- to picoamps. There's leakage to account for too as features get smaller and smaller due to tunneling and other effects, but still in aggregate the current per transistor is tiny.
Also the transistors are working at 1V or lower, but as you say they are FETs and don't have the same Vbe drop as a BJT.
You are right, I mixed this up. If you take a CPU running at 100 W with 10 billion transistors (not quite realistically assumed to all be wired in parallel) at 1 V, you would get an average of 0.01 microamps. So the factor would reduce to roughly 10^5.
Wait a minute, a lot of those transistors are switching the same currents since they are in series. Also, FETs only draw most current while switching, so in between switches there's almost no flow of electrons. So in fact you cannot calculate things that way.
Yes, as I said the parallel assumption is not quite realistic, and the number is an average, covering all states a transistor may be in. So it amounts to a rough lower bound for when a transistor is switching.
Thank you for this - the name neural networks has made a whole generation of people forget that they have an endocrine system.
We know things like sleep, hunger, fear, and stress all impact how we think, yet people want to still build this mental model that synapses are just dot products that either reach an activation threshold or don't.
Fortunately for academics looking for a new start in industry, this widespread misunderstanding has made it only far too easy to transition from a slow-paced career in computational neuroscience to an overwhelmingly lucrative one in machine learning!
There have been people on HN arguing that the human brain is a biological LLM, because they can't think of any other way it could work, as if we evolved to generate the next token, instead of fitness as organisms in the real world. Where things like eating, sleeping, shelter, avoiding danger, social bonds, reproduction and child rearing are important. Things that require a body.
Isn’t that taken the analogy too literally? You’re saying nature is promoting humans to generate the next token to be outputted? What about all the other organisms that don’t have language? How do you distinguish nature prompts from nature training datasets? What makes you think nature is tokenized? What makes you think language generation is fundamental to biology?
I would imagine the baseline assumption of your thinking is that things like sleep and emotions are a 'bug' in terms of cognition (or at the very least, 'prompts' that are optional).
Said differently, the assumption is that with the right engineer, you could reach human-parity cognition with a model that doesn't sleep or feel emotions (after all what's the point of an LLM if it gets tired and doesn't want to answer your questions sometimes? Or even worse knowingly deceives you because it is mad at you or prejudiced against you).
The problem with that assumption is that as far as we can tell, every being with even the slightest amount of cognition sleeps in some form and has something akin to emotional states. As far as we can prove, sleep and emotions are necessary preconditions to cognition.
A worldview where the 'good' parts of the brain (reasoning and logic) are replicated in LLM but the 'bad' parts (sleep, hunger, emotions, etc.) are not is likely an incomplete model.
Ah a very fun 'snippy' question that just proves my point further. Thank you.
No airplanes do not sleep. That's part of why their flying is fundamentally different than birds'.
You'll likely also notice that birds flap their wings while planes use jet engines and fixed wings.
My entire point is that it is foolish to imagine airplanes as mechanical birds, since they are in fact completely different and require their own mental models to understand.
This is analogous to LLMs. They do something completely different than what our brains do and require their own mental models in order to understand them completely.
Here, it's actually fun to respond to your comment in another way, so let's try this out:
Yes, sleep is in fact a prerequisite to planes flying. We have very strict laws about it actually. Most planes are only able to fly because a human (who does sleep) is piloting it.
The drones and other vehicles that can fly without pilots were still programmed by a person (who also needed sleep) FWIW.
It works because well, its actually pretty primitive at its core. Whole learning process is actually pretty brutal. Doing millions of interations w/ random (and semi-random) adjustments.
Honestly once you understand maximum-likelihood estimation, empirical risk minimization, automatic differentiation, and stochastic gradient descent, it's not that much of a surprise it works.
Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range...
That seems implausible. Apple's M2 has 20 billion transistors and draws 15 watts at full power [1]. Even assuming that 90% of those transistors are for cache and not logic, that would still be 2 billion logic transistors * 1 milliampere = 2 million amperes at full power. That would imply a voltage of 7.5 microvolts, which is far too low for silicon transistors.
The more of our logic we can implement with addition, the more can be offloaded to noisy analog systems with approximate computing. It would be funny if model temperature stopped being metaphorical.
A simple discretization of the various levels of signal at each input/output, a discretization to handle time-of-propagation (which is almost surely part of the computation just because it _can be_ and nature probably hijacks all mechanisms), and a further discretization to handle the various serum levels in the brain, which are either inputs, outputs, or probably both.
Just add a factor 2^D transistors for each original "brain transistor" and re-run your hardware. Hope field effects don't count, and cross your fingers that neurons are idempotent!
Easy! /s
Modelling an analog system in digital will always have a combinatorial curse of dimensionality. Modelling a biological system is so insanely complex I can't even begin to think about it.
One important reason for the difference in current is that transistors need to reliably switch between two reliably distinguishable states, which requires a comparatively high current, whereas synapses are very analog in nature. It may not be possible to reach the brain’s efficiency with deterministic binary logic.