I'll bookmark both those papers - and I rarely bookmark anything (instead relying on Google and the eternal onslaught of ever newer links and issues).
That paper is real fun to read, an excerpt:
> How would we begin? First, we would secure funds to obtain a large supply of identical functioning radios in order to dissect and compare them to the one that is broken. We would eventually find how to open the radios and will find objects of various shape, color, and size (Figure 2). We would describe and classify them into families according to their appearance. We would describe a family of square metal objects, a family of round brightly colored objects with two legs, round-shaped objects with three legs and so on. Because the objects would vary in color, we would investigate whether changing the colors affects the radio's performance. Although changing the colors would have only attenuating effects (the music is still playing but a trained ear of some can discern some distortion) this approach will produce many publications and result in a lively debate.
If you need a real good laugh, do read it now, it gets even more funny after the quoted paragraph! It's almost as good as that (much shorter) paper on PubMed where they examined the sterility of farts in operating rooms (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1121900/).
One night I had a dream a biologist not knowing what a computer is would spend hours describing that mysterious non uniform squarelet mater engraved with latin letters. What a curious organ.
Laugh all you want, but if ever an UFO crashes on Earth and we find a mysterious alien computer, we will need people with that mindset to reverse-engineer it... ;).
>"Even if a diagram makes overall sense (Figure 3A), it is usually useless for a quantitative analysis, which limits its predictive or investigative value to a very narrow range. The language used by biologists for verbal communications is not better and is not unlike that used by stock market analysts. Both are vague (e.g., “a balance between pro- and antiapoptotic Bcl-2 proteins appears to control the cell viability, and seems to correlate in the long term with the ability to form tumors”) and avoid clear predictions."
Exactly! As soon as you start trying to attach numbers to this stuff you realize we haven't figured out much of anything about it, and these "theories" haven't ever really been tested. It is honestly a waste of time to approach the problem like that. Even assuming the info is accurate (yet vague), future generations will have to redo all of it anyway to see what is is really going on. In reality it seems to be both inaccurate and vague...
One lesson I took from that paper was just how immature the modern software industry is. If you ask a software engineer to describe a system graphically they'll produce diagrams that look like the biologist's, not like the electrical engineer's. There's a general lack of standardized, accurate, and useful terminology or even models to describe the high-level structure of software. The best we have are a few analogies, some described design patterns, and low-level data structures and algorithms. We can describe things on the level of gears or resistors but we don't have a good vocabulary for things on the level of transmissions, engines, ALUs, or switch-mode power supplies.
Worse, we don't seem to care a lot about this lack, largely because it isn't perceived. A lot of devs think that we're in the space age when it comes to software, instead of virtually the stone age. And then we wonder why our software tends towards the "big ball of mud" model oh so often.
I agree that software lacks this visual language -- I always find English a better architectural description language than UML, though I am not particularly happy with either.
I think the reason is that computer programs are documents -- things constructed in a language. Someone might conceivably write a good summary of, say, _War and Peace_, but a diagram of the book would only be useful for illuminating certain aspects of the story.
Describe in, say, five thousand words or less the architecture of nginx in a way such that if implemented it would have demonstrably the same characteristics as the existing version.
The reason why we use words like 'web server' is so that we don't need to carry around a 5000 word description of a 'web server'.
Just as when building a car one would use a word like 'engine' rather than carry around a 5000 word description of an 'engine'.
By compressing highly complex machines into single words we're able to build much large machines by treating that complex machine as a single component of the much large machine.
Furthermore, I fail to see how whether one can describe nginx in 5000 words or less means that our industry lacks the vocabulary to describe components of more complex machines built from those components.
If your developer requires a 5000 word description of nginx to type
apt-get install nginx
then they are a tool and need to be fired.
Most of the problems with our industry stem from exactly this kind of problem, rather than look at the vast array of components that do the exact same thing they are about to implement, they use something like nginx to make nginx, except instead of installing nginx they made a 5000 line nginx config file to reimplement most of what nginx already does.
If you've ever seen
{ "key" : "23", "value": "Michael Jordan" }
then you know what I'm talking about in regards to the stupidity of this industry using components to make the component it already is.
This question is like when someone says they are intelligent then someone gives them a wrote memorization test about the agreed upon symbols we use to encode those words on paper. While memory is required for intelligence, not having memorized a particular encoding of a particular word is not an indication that one does not possess intelligence.
If one wrote such a description, it would form the source code for nginx. The source code for a program is a description of the program's architecture, in all of its details.
Do not mistake the source code, for the program itself! The documents programmers spend all our time working on are really just specs; the actual program is an artifact produced by a compiler.
This isn't entirely true, you could compress the source code which would generate a dictionary (vocabulary) which could be used with the description (compressed output) to recreate the source code.
You see comp sci is so advanced we don't even bother writing descriptions of our creations, we allow the software we've written previously create those descriptions for us, along with the dictionary used to understand the description, and then use another program to understand that description for us.
That essay is fun but supremely unfair to biologists. The reason that we engineers can "unambiguously understand a diagram describing the radio", "identify familiar patterns or modules in a diagram of an unfamiliar device," or "[use a] formal language that successfully unites the efforts of many individuals" - is because we created all of those things from first principles. We - people - invented radios and all of the parts in them, of course we know how they (and other man-made devices) work. The engineer doesn't have some special mindset that allows him to fix the radio. He just has a lot of knowledge about how the system works, because it was written down by the people that created it, all the way back to simple combinations of common matter.
It's easy to have a formal language and tools when you start with the language and build everything on top of that. When you're trying to look at a complex system that was entirely built by some unknown process, according to unknown rules, and you have to figure out the language from observation, it's obviously going to be a very different ballgame.
What a fun approach. It's true that traditional neuroscience methods are too crude to give insight to brain functions and results are often over-interpreted. Neuroscientists know that however [1], and that's why attempts to simulate large circuits like the blue brain project or the simulated cat model produced nothing more than "oscillation statistics". Despite their crudeness, there are cases where circuits (like the amygdala) were causally linked to behavior by early electrophysiology .
There is hope however, as there are now optical and molecular methods that make it possible to observe , activate, and inactivate individual neurons, which allows making causal inferences[2].
This paper doesn't describe "traditional" methods. The author test analyses used today. The first simulation specifically tests whether inactivating "cells" (components in the microprocessor) can let us make causal inferences. They conclude that it can't. Whether that's a fair conclusion is up for debate, but they are definitely critiquing the most cutting edge techniques in the field
Couple that, however, with the ability to tag the cells participating in an experience, the ability to specifically reactivate them, and the ability to correlate them with other experiences. Their analogy in general has a big flaw, in that we know already that the brain does not have the serial logic of a circuit. The output of the brain is also not predetermined behavior, but rather dynamics between large, sparse populations, which somehow combine in interesting ways. This problem may be interesting to study in a more rigorous theoretical manner, but i don't think their approach may be taken seriously as an indicator of anything.
What "traditional" methods are you referring to? Hubel and Wiesel mapped the functional anatomy of the visual cortex in the 1950s by poking at cat brains. We've learned more since but their original model is still largely correct.
+1 for a cool and thought-provoking idea but -1 for more sloppy/faulty reasoning and arrogance than could be overlooked by a sincerely interested reader.
I am sympathetic to the angle this is coming from. Yes, the field has a serious problem with shoddy statistics and post-hoc theories optimized for sexiness. But this paper vastly oversimplifies the problem domain
to the point of trivializing it.
The approach of this paper is to take techniques from a largely mysterious domain, apply them to a vastly different domain that we understand much better, and conclude that because we can't replicate our prior understanding of the known domain there must be a problem with the tools. My guess is that this approach looks less ridiculous on the surface because of how entrenched the brain-computer metaphor is--but don't forget it is just an anology: a knowingly incorrect abstraction. As the authors admit, lesioning a circuit board is totally different from lesioning a brain. So why on earth should we be talking about how much we can learn from brain lesions based on our understanding of microprocessor lesions? (not a rhetorical question--I am genuinely curious if I am missing something here because this absolutely confounds me)
Interesting paper, thanks! I'll need to work through it more at a later time, because I got stuck at the Lesion studies argument. I'm sorry if this seems a bit ranty..
As for lesion studies, I wonder why there was no mention of single and double dissociation? Only single dissociation was discussed, and although it is tempting to say lesioned area X is uniquely responsible (a combination of necessary and sufficient) for behaviour Y, it only tells you something about its necessity, and nothing about its sufficiency. This is generally known in Psychology & Neuroscience, which is why almost all studies base themselves on double dissociation. It allows you to say sensible stuff about differences between behaviours at the same level of description (Space invaders vs Donkey Kong being different, for example), due to differences in a lower level of description.
Also, I feel like the importance of topology is somewhat disregarded in discussion of these lesion studies? As we go lower down in levels of description - systems, to clusters, to individual neurons or even molecules, topology of the units of description becomes more and more important. Imagine enumerating every transistor on the chip and switching their connectedness around to the point where behaviour Y can still be performed, but lesioning transistor X no longer makes behaviour Y impossible. So we don't actually obtain a "large number of identical radios", we obtain a large number of functionally identical, but topologically different, radios, and then shoot metal particles at them to understand under which conditions area X does produce behaviour Y. This, to me, seems a lot less futile and a lot more informing then what the section on lesion studies leads me to believe?
I also noticed the lack of a "double dissociation" study. Fwiw there's a ton of studies right now using optogenetics to transiently "lesion" areas where the authors have not done double dissociation type experiments to determine the specificity of their effects. My guess is the authors were in part critiquing that
"The fundamental unit of biological information processing is the molecule, rather than any higher level structure like a neuron or a synapse; molecular level information processing evolved very early in the history of life."
http://www.softmachines.org/wordpress/?p=1558#more-1558
I assume there are some neuroscientists here. I was trying to imagine how the brain thinks, and how it comes to conclusions and taps into a breadth of information so quickly.
One way I tried to conceive of it was that when the neurons in your brain fire, they compose patterns. These patterns -- the orders and timing of neurons firing -- might be likened against something like a hash table, wherein you represent data as a serialized pattern.
For instance, when I think of a dog, my brain fires some base neurons that are associated with the size of a normal dog, and some of its most basic attributes: 4 legs, fur. These could also be the same regions of the brain that would fire when I think of a cat, or a raccoon.
Does this in any way represent how the brain actually works?
I'm a computational neuroscientist (and data scientist and software engineer and startup founder).
The general principle in the brain is that the individual neurons (there are 25 billion in the cerebral cortex) represent statistically frequent occurrences of experienced phenomena. So if you know a lot about dogs and interact with dogs a lot, you will have a lot of neurons that "represent" different things about dogs (dog categories, dog behaviors, aspects of dog appearances, etc.).
The neurons "represent" perceptual experiences in a collective way called a population code. In one study on humans, a neuron was found that fired when viewing pictures of Jennifer Aniston but only when Brad Pitt was not in the photo. This does not mean the neuron had the sole job of representing Jennifer Aniston, but only that it was "tuned" to this perceptual occurrence. The tunings of neurons are distributed to "cover" in a statistical fashion the range and components of experiences. This particular human subject had perhaps watched many episodes of Friends.
What is still unknown is how the structure of a visual scene is represented. Neurons have been found for edges, contours, shapes, motion, depth, and objects. The mystery is how they all work together to parse and compose the scene. This is hard to determine because it is usually only possible to "listen" in on a few neurons at a time with electrodes, whereas it takes hundreds of millions to represent the visual world.
Can I float a conceptual conversion by you? I'm a pathologist (I look at human brain regularly and interact with neurologists, neurosurgeons, and neuropathologists) and my undergrad is in physics. I like your general representation. Here's my conversion:
Let's start with a large matrix operation, something a deep neural net neuron would do. Let's imagine that matrix is a sheet with colored dots instead of numbers.
We don't care so much about the order of the rows, but the direction of the columns holds some meaning. So we can connect the top of the matrix to the bottom, like a tube. Now the left side isn't exactly a beginning, and the right side isn't exactly the end, but there's this sort of polarity.
These ends of our tube (which was a sheet) are rings, and those rings can be reduced to points, something like Grothendieck's point with a different perspective at every direction (or in this case, many directions, one direction per row). But the left point and the right point are still different.
Now I have a polarized bag.
Like a neuron.
I could be silly and imagine the gates and channels on the neuron surfaces could be mapped to elements in the matrix like those colored dots, but I rather doubt the analogy extends quite that far...
And neurons don't absorb many giant tensors and emit one giant tensor. But they do receive their signals at many different points on the surface. So there is this spatial element to it. And there are many different kinds of signals (electrical, neurochemical, all the way to simple biochemical, like glucose). So there's this complexity that an inbound tensor would represent nicely. And they do in fact emit a single signal sent to many neighbors.
Anyway, that's my personal matrix-to-neuron conversion.
It feels like a wrong analogy - the large matrix operations are not really want a deep neural network is doing, it's an implementation detail, an artifact of how it's efficient to represent a large number of neurons and their connections.
The results of those tensor operations (not in their total, but each particular output number) may have some very rough analogy to the changes in a particular single synapse "connection strength" caused by various biochemical factors as a result of neuron operation, but the whole tensor operation doesn't map to any biological concept smaller than e.g. "this is a bit similar to how a layer of brain cells changes its behavior over weeks/months of learning". A machine learning iteration updating all of the NN weights is a rough correspondence to the changes that, over time, appear in our brains as a result of experience and learning (and "normal operation" as well).
I have seen an interesting hypothesis on how the particular layout of a dendritic tree and it's synapses can encode a complicated de facto "boolean-like" formula out of all the "input" synapses (e.g. a particular chain of and/or/not operations on a 100+inputs), instead of essentially adding all the inputs together as much of artificial neural networks assume, but I'm not sure about if how such hypothetical "calculations" actually are used in our brains.
I certainly wouldn't argue with neuroscientists and network experts here, but as a clinician dealing with products of brain function, that is, thinking, problem solving, sensory phenomena, etc., what happens in brains has long been a subject of intense interest and study.
I have the idea what gets lost in the various hypothetical models of neuron/brain mechanisms is the sheer magnitude of neuronal circuit complexity. Not only are there a vast number of "nodes" in human brain circuits, something like 100 trillion synapses, but that each node/synapse is itself enormously complex.
It's hard to quantify the number and variety of receptors "owned" by each neuron, but consider that while neurons work by communicating with certain "partner" neurons, they also are connected by very complex channels to all parts of the body. Furthermore neuron signal transmission involves the cascade of intracellular "downstream" effects mediated by intricate genomic events.
We know something about some of these channels, others not so much. "I/O" occurs through nerves, hormones and immune system signalling. In isolation each of these systems is highly complex, but considering the nature and meaning of interactions among these systems is overwhelming.
Given the immense scale of complexity it's unsurprising that we have only a primitive understanding of how neuronal systems work. I don't anticipate science will "figure it out" in my lifetime.
OTOH no doubt science will continue to discover intriguing clues in the coming decades, and that will be valuable. But I think it's important to appreciate just how vast the "problem space" is when it comes to grasping how brains actually work.
You may very much enjoy the book "On Intelligence" by Jeff Hawkins. He was the founder of Palm computer with a strong computer science background and then went on to found his own neuroscience institute.
He has been working on AI in a system which he terms Hierarchical Temporal Memory which he describes in his book. The interesting point is that his algorithms and data structures are based on existing neuroscience research and "reverse engineering the neocortex". It sounds very buzzwordy but I found the book an excellent cross-disciplinary read.
Yes, this is roughly how it is believed to work. In fact, one can make out neurons that are reliably involved in representing concepts like location and orientation [1], places [2] and persons [3]. Usually, multiple neurons are involved in representing single concepts (population coding [4] and distributed representations [5]). Concepts are constructed by successively extracting features and combining simpler features in more complex ones, from the sensory input regions towards the association regions in the cortex (see feature hierarchies [6]). You can visualize thoughts as population-rate coded probability distributions that interact very quickly in single "torrents" of activations.
You can find a neuron that fires when you see your dog. But whether it is a general dog detector, tuned to just to your dog, or part of a distributed sparse encoding etc. is not conclusively known
One of the major outstanding questions I have as a neuroscientist is whether the classic experimental approaches used here can ever get us to the 'understanding' the is needed to build something that looks like a brain. I have been leaning toward the thought that we are likely to make faster progress by letting the synthetic biologists have a shot at it, even if it means we will be stumbling around in the dark if we stray even slightly from the steps they learn to take.
Very insightful article. I will try to "cell it" to our next Journal Club when we will be talking about the latest thinking on Brain-Computer Interfaces. 8-)
This is an interesting idea, and the paper is pretty well thought out. But I think one source of information not sufficiently explored is anatomy, which would help a great deal with the microprocessor, although it seems to help less with the brain. If you have the connectivity of the entire microprocessor (as the authors have determined using microscopy), then you can probably determine that there are recurring motifs. If you can figure out how a single transistor works, then you can figure out the truth tables for the recurring motifs. That takes care of most of Fig. 2. The only question that remains is if you could figure out the larger scale organization.
Anatomy also helps with the brain, but not nearly as much. People are still trying to figure out how the nematode C. elegans does its relatively simple thing even though the full connectome of its 302 neurons was published 30 years ago. But in larger brains, the fact that neurons are clustered according to what they do and projections are clustered according to organization in the places that they project from provides at least some level of knowledge. We are not merely applying Granger causality willy-nilly to random neural activity. We know what's connected to what (at a gross level) and in 6-layer cortex we even have an idea of the hierarchy based on the layers in which projections terminate (which is how Felleman and Van Essen got Fig. 13b in the paper).
OTOH, I think our failure to understand many neural networks at a conceptual level is quite disturbing, and perhaps a sign that the kind of conceptual understanding we seek will be forever beyond our reach. The authors mention this toward the end of the paper, although I think they overstate our understanding of image classification networks; I've never seen a satisfying high-level conceptual description of how ImageNet classification networks actually see. One possibility is that we simply don't have the right concepts or tools to form this kind of high-level description. Another possibility is that there simply is no satisfying high-level way to describe how these networks; there are only the unit activation functions, connectivity, weights, training data, and learning rule. We can find some insights, e.g., we can map units' receptive fields and determine the degree to which different layers of the network are invariant to various transformations, but something with the explanatory power of the CPU processor architecture diagram in Fig. 2a may very well not exist.
I hope that the brain's anatomical constraints provide some escape from this fate. Unlike most convolutional neural networks, the brain has no true fully connected layers, and this may serve to enforce more structure. We know that there is meaningful organization at many different scales well beyond early visual areas. At the highest levels of the visual system, we know that patches of cortex can be individuated by their preferences for different types of objects, and similar organization seems to be present even into the "cognitive" areas in the frontal lobe. It remains to be seen whether it's possible to put together a coherent description of the function of these different modules and how they work together to produce behavior, or whether these modules don't turn out to be modules at all.
Author here -- so actually we've done a fair bit of anatomical work recovering motifs, even while looking at the processor [1] -- we took that out of this paper based on several readers' recommendation that the content was "too new" and that people wouldn't understand it. That said, without the ability to then directly probe those specific circuits, it's quite hard to figure out what the motifs are doing, and motif finding in general is an incredibly challenging problem, especially at scale.
If you can figure out how a single transistor works, then you can figure out the truth tables for the recurring motifs. That takes care of most of Fig. 2. The only question that remains is if you could figure out the larger scale organization.
I sure hope a brain does not have as many abstraction layers as a microprocessor (there are half a dozen of such layers between truth tables and applications). I think that it would be much harder to reverse engineer a microprocessor than a human brain if you're equally clueless about them.
My guess would be that they might make progress in the direction of physics but not in the direction of higher order abstractions further up the stack. Those would just be interpreted as rules of the universe or background noise.
Your wife performs (or, can perform) arbitrary experiments on classic microprocessors to see if popular data analysis methods from neuroscience can elucidate the way they process information?
