A lot of this seems to be a problem with academic publishing, peer review, and the pedantry contained therein.
I tried (twice!) to publish a paper that was scientifically sound, but written such that it could be understood by a lay audience. It was rejected; quoting a reviewer: "This paper lacks math." That sentiment made me lose a lot of faith in academia. Now, I simply refuse to contribute any more time to writing papers or performing peer review (esp. in non-open publications). I'm sure I'm not alone; I know of at least one "seminal" robotics paper that was rejected from top venues multiple times for "simplicity" (lack of mathematics content) that went on to become a foundational paper in the field after appearing in a lower-tier venue years later.
The irony: it takes researchers a lot of time to make a paper dense & concise. If they "showed all steps", it would probably improve researcher productivity & make the material more approachable to newcomers. Instead publishers enforce length restrictions... for which authors dedicate upto 25% to related work (some of which is useful; much of which is pandering to likely peer reviewers in small, niche fields). Length restrictions seem equally foolish in the age of digital publishing. And again, inadvertent pedantry is the only explanation I can imagine... but happy to be wrong.
This may be intentional - for most academic publishing venues, their target audience is other researchers of that domain. It's counterproductive to optimize an article for a lay audience if everybody in the actual audience has (or is working on) a PhD in that domain and is generally expected to be (or become) up to speed with the terminology and earlier results on that topic. The journal or conference is generally not meant to communicate their results to wider audience, it's generally a tool that some particular research community made for themselves to help their research by an exchange of ideas and results. An article in academic publishing is meant for someone who will use that to further their own research in a related field - and the needs of a person like that are very far from a lay audience, they will want to see entirely different things in that paper - you want the paper to focus on the novelty, on the "delta" between this result and what was bleeding edge a few months ago; not spend two thirds of the paper describing what was already known before.
There are publishing venues intended for a lay audience, but most academic publishing is not, they have incompatible goals.
Unfortunately, the 'lay' audience is often not all that 'lay.' For example, you might be working on a phd in an adjacent field to a given paper and still have a hard time coping with the sub-field-specific jargon. This happens all the time in math research, and contributes heavily to silo'ing of subfields. This silo'ing is actually super counter productive, and can lead to the same things being independently rediscovered, and important, useful discoveries simply failing to be communicated across silo boundaries.
I'll echo this for biology too. Often, the jargon is the exact same word, but used in a very different way. A quick example is the word 'vector'. In math, a line in space, in genetics, a vector is a virus. There are many other examples, especially with proteins, where those acronyms are reused many times, sometimes inside of the same article.
I was talking to a student the other day. He and I took many of the same classes in different semester in which we were assigned to read articles and then present them as part of class discussion. Typically this ends up being a talk all about the figures of the paper. He mentioned that most of the time, with bio papers, you have to read the article at least 3 times before you can start to grasp what is going on with it, not to mention the online methods section that goes on for ~20 printed pages. Often, the figures are quite clever and really do a good job at conveying the data in a great way, but they are presented as 'in-field' and the graphs, the diagrams, the data, etc. will be very very difficult to understand from a 'novice' point of view. For example, they will present the data out in a polar plot where the 360 graph is known in the field to represent only the arc of sensation out of the right 7th whisker of male mice under anesthesia. In that (contrived) example, you really have to dig about in literature to find this out and if you don't know that then you'll end up confused and really frustrated. Essentially, if you are a person that is somewhat interested in a very closely field of research, you have to spend hours upon hours trying to parse out some piddly little paper. It really is that bad. I don't know what that is, but it sure ain't 'science'. A paper in a closely related field should take you no more time to read and understand to the point of being able to tell someone else than the time it takes to eat a sandwich, a bag of chips, and drink a coffee in total.
Oh man, check out youtube or every other podcaster for that. Like, millions of videos of people 'talking' about papers, though they run into copyright stuff all the time too.
One thing I've learned is that someone in that position should be looking at PhD theses before journal papers. They can be long enough that they function much better as introductions.
It's a sad industry that has enshrined culture that the more experienced/senior a person is, the lower quality their work is, because they have to make it artificially inaccessible to make themselves appear more elite.
This and review articles are the only two avenues currently available. I'm glad to see the original post here making another effort to provide resources.
I agree with your basic premise: Researchers are not supposed to be writing for a lay audience. No arguments here.
But too often, the language (both english & mathematical) of research papers is not intended to enhance academic communication. Rather, it's a faux optimization to game peer review or make results look more "technical" in a way that is actually detrimental to communication.
In effect, the language is counterproductive. And the time spent on faux conciseness is wasted; much like that old quote, "I would've written a shorter letter, if only I'd had more time."
Out of curiosity, which field are you working in precisely?
In my experience (in type theory / programming languages), the main problem is that a lot of papers are just badly written. Good papers really do use precise language to be easier to digest for other people in the same field.
The reason why researchers write for a niche audience is because they are maximizing publication count. We're essentially targeting the papers for their reviewers. If researchers did not care about publication count, they would target the widest possible audience so that their results would have the most impact.
It's rather the other way around, it's that reseachers read only/mostly works designed for a niche audience; and thus only value such works. We measure "the most impact" as "the most impact on us", i.e. how much the work impacts the research community.
Works have an impact if I read them - a publication that targets a wide audience and thus by necessity is three times as long and "too introductory", is a publication that won't be read (by other researchers), won't be cited (by other researchers), and thus won't have an impact. If lots of laypeople read and understand it, well, that's nice, that's how textbooks work, but it doesn't help advance the science. A citation in mass media doesn't count as a citation. If someone silently adapts your method in industry, that's nice, but it doesn't really matter as well - that's a sign that your tech is mature, but "impact" measures the impact that your paper has on future progress - if the next thing doesn't rely on your paper, it doesn't have an impact on your research field anymore. Reviewers are actually representative of the intended audience of readers, so targeting for people like them makes sense.
"Maximimizing publication count" seems to be the accepted ground truth, but I'm still somewhat surprised researchers don't write for a larger audience because one would think that expanding your audience beyond your sub-field will have a higher chance of garnering citations.
Maybe the "lottery" citation factor isn't worth the extra effort required to normalize in-field shibboleths. Then again, if maximizing citations truly is the overriding publication incentive, I wonder why low h-index researchers don't pursue this route and 'upgrade' their existing papers for a wider audience wherever possible.
And perversely, this makes it so that people feel compelled to add mathy things that add no additional explanatory power to the results.
I've frequently heard complaints that people in some CS subfields in particular will just add complicated math because they have to rather than because it helps. It sucks because the authors don't have that great a grasp on the math anyway, the readers don't care about the math -- they just care about the intuition and process, and communication suffers overall.
My best papers had at most like two equations, and they were just there to clarify models for fits and the like in case the reader wasn't familiar with terms like "Arrhenius relation". So, you know, to actually make reading it less burdensome rather than just making it look more sophisticated.
Exactly! In the end, we ended up just publishing on arXiv.org and writing up a blog post [1]. It's not earth shattering... just an entirely new idea being demonstrated for the first time. The mathy background certainly exists, and was explained in other cited papers, but wasn't necessary to explain the new ideas.
I've definitely had reviewers tell me in a paper that I needed to make my math more formal and really prove things, when all I really wanted to say was, "We take the limit of foo as bar goes to zero, and get baz. Look, baz is interesting!"
Here are my thoughts based on my experience with peer reviewed publication.
There are two high level criteria for publication: novelty and difficulty (this is in my field of Programming Languages and Systems so keep that in mind).
The novelty requirement is important and I trust that you satisfied it but (as you pointed out in a child comment) you may not have met the difficulty requirement and the reviewer did their "best" to articulate that in a way that isn't the all together ridiculous "not hard enough".
Naturally we might wonder why "difficulty" is a requirement at all. Shouldn't the importance or impact of the work be the thing that matters regardless of how difficult it was to achieve? The problem is that it's _extremely_ hard to know what work will be impactful and so reviewers, who have to reject something like 90% of submissions, use the heuristic of "difficulty" to estimate.
This is a problem to be sure but I think it would be a problem in other settings as well.
I wanted to echo this experience. I agree that reviewers are generally looking for novelty and difficulty. Unfortunately, difficulty can translate into a system where reviewers create small fiefdoms: This isn't difficult enough for my journal, so I'm going to reject it. I say fiefdoms because I don't often see criteria for what difficult means, so it's left to individual reviewers and the editor to decide. Anyway, this creates some perverse incentives where it's now necessary to make a piece of work appear more difficult than it really is.
That said, I don't really think this is all that unique to academic publishing. At some point, I think we've all met someone professionally who makes problems sound more difficult than they really are and seen this person rewarded by management because they're the person who can work on really difficult problems.
I'm more on the theory end of PL, but in my experience program committees generally regard difficulty as a negative for a paper. The main thing PCs want is for papers which make progress on important problems. Now, you can do this either by (1) attacking existing open problems, or (2) by finding new important problems.
It's easy to tell that a long-standing open problem is important, but generally these problems are open because they are hard, and consequently the technical sophistication for new attacks on them is generally very high.[] New problems can often be solved with relatively simple techniques, but it's also difficult to tell if a new problem is genuinely important or just some random thing made up for submission.
With a low acceptance rate, PCs tend to favor type (1) papers over type (2) papers, because few people feel comfortable arguing for a paper that might* be doing something important over one that is doing something important. This is widely seen as a failure mode of low acceptance-rate venues.
So over the last several years many of the top PL conferences (such as POPL, LICS and PLDI) have moved to a multiple track format in order to increase the number of papers they accept. This is precisely to give program committees the freedom to take chances on "risky" papers that might become important, or might become footnotes.
[*] Occasionally, someone will solve a difficult open problem with simple techniques, and these papers are immediately accepted with much rejoicing. It's easy to have your cake and eat it, when you have two cakes...
> in my experience program committees generally regard difficulty as a negative for a paper.
I don't know if we're talking about the same type of difficulty. I don't think program committees see the difficulty of the problem being solved at the core of the paper as a "negative" necessarily?
> The main thing PCs want is for papers which make progress on important problems. Now, you can do this either by (1) attacking existing open problems, or (2) by finding new important problems.
My earlier note aside, I think this is an extremely important point. It seems to support the idea that "impact" is hard to gauge but much easier when the community has already arrived at some form of consensus on the importance of a problem.
I don't know about the details of the OP's paper, but I'm not sure "difficulty" is the right word... Mathematical rigor is needed in many domains. In my field, "not enough math" could well signify that the description of the experimental results is sloppy, or that you can't check whether the results presented are sound.
> A lot of this seems to be a problem with academic publishing, peer review, and the pedantry contained therein.
Ayup. It's too annoying to translate the academic speak, but you can shortcut it.
For example, anything CS nowadays (I include crypto in this) must be 1) publicly available on the web, 2) come with running code that runs some basic tests, and 3) be limited to a single compilable file. Failure in any of the criteria means I move on. If the source code involves anything like "configure" it's an immediate fail. If the code passes all three of those criteria, it's probably more useful to read the code rather than the paper.
I used to love ISSCC for the reason that they used to demand both A) a die photograph of a chip and B) actual oscilloscope traces. You can't hide when you have to make test equipment produce data. Sadly, they got rid of that requirement in the late 90's, and the information content of the conference suffered correspondingly.
Not as often as you think. And there are multiple levels to this.
1) CS papers are normally about a single or logically-related set of concepts. If you can't reduce the code to a single file, the paper is probably too broad or you are trying to hide something.
2) If the author wants me to take time, it has to be really easy to try. "Configuration" is asking me to make choices that I really have no clue about a priori. Now, maybe there is a "heavy duty" version as well. But the "straightforward" version needs to be really straightforward.
And you would be surprised how often you don't need the heavy duty version. A nice example of this in crypto is DJB's "TweetNaCl". It gets you 99% of what you need until you understand what it doesn't get you. And, it's so damn easy to include that you get very little resistance--try using tweetnacl vs libnacl vs libsodium in a project. You'll pull your hair out if there isn't a magic package already for libsodium while tweetnacl can be dumped into embedded microcontrollers without pain.
3) If the author wants longevity, simplicity is his friend. Can you even begin to run a "configure" script from even 5 years ago let alone 10? Yet, I can download an ancient Fortran 77 code file and compile it today pretty easily.
If what you write is difficult to understand people will assume you're smart. If you've ever learned a foreign language you'll recognize this: Blithering idiots talking in that language sound smart because you're having a hard time understanding them.
In my experience, papers often take more than just two submissions to be published. It's often a question of finding a good community fit. One paper I coauthored took 8 years and probably 10 submissions before it was finally accepted.
Length restrictions can be a bummer, but if you have a publishable result that you simply can't squeeze into 10-20 pages (depending on the venue), typically you split it in to two or more publications. This has the added advantage of ensuring that each published unit is a smaller, tighter piece of work. I don't think it's just pedantry.
Many people would say that if you submitted 10 times, you are playing the system. You've had 10 sets of reviewers spend time to review your work, even though 9 of them told you that it's not worth publishing.
I guess that's the advantage that tenured researchers have over everyone else--you can't run out of career before you published all of your papers.
I don't really agree with the "playing the system" sentiment. It isn't as if we took the same paper 10 times and just submitted it until it stuck somewhere. Each time we got valuable, actionable feedback, which we then acted upon; the final paper was almost unrecognizable from the first submission except in the core ideas. We gained a lot of experience as a team from this process, and the end result is a much, much better unit of work entered into the literature.
None of my other publications have taken 10 submissions, so this case is also probably an outlier.
Depends on what reviewer actually told you. There is not worth publishing and there is not worth publishing in this journal. Those two are not the same.
Moreover, there is level of subjectivity in peer review.
Carl Sagan was famously denied tenure at Harvard and membership in the National Academy of Sciences for his science advocacy. Different takes on it range from "jealousy" of his peers over his broader popularity to anger from them for his making science accessible and explaining concepts in a way that allowed readers to understand them:
I used to see this with my peers in computer programming in the 1990s. There was a lot of anger and jealousy when everyday normal people started putting up websites. Several of my CS friends were of the opinion that this was almost polluting the WWW with bad code.
(Also: The mathematical underpinnings are related to radar systems, whereas the application is in robotics. The amount of background information to bring a roboticist up to speed would've consumed entire papers or books -- some of which were previously published by myself & other coauthors.)
In my field (physics) I find that an increasing number of journals allow supplemental materials. You provide the main story in the text, but can add a lot in the supplemental materials. For example, suppose we have discovered a new material with some exciting properties. There is a fair amount of characterization that we do just to convince ourselves that we made what we think we made. A lot of this can go into the supplemental materials for those that want to make sure that we have done the standard sanity checks. A long time ago, you would write a short letter, then a longer report. That fell away, but I am really a fan of supplemental materials when used correctly.
"I know of at least one "seminal" robotics paper that was rejected from top venues multiple times for "simplicity" (lack of mathematics content) that went on to become a foundational paper in the field after appearing in a lower-tier venue years later."
Correct me if I'm wrong, but if you don't value what a publisher is offering, you could publish your paper on your own, make it accessible to an audience of your choosing with a delivery of your choice (for example, the wonderful work of Aphyr, here: http://jepsen.io/).
I don't get why people would try to publish in high-tier venue. To me it seems much more about polishing one's ego instead of improving the research quality.
As a young researcher, you need the publications when you apply for academic positions. High-tier venue publications are easily recognized as such. To recognize high-quality work published in a low-tier venue, the only metric is citations. Accumulating citations takes a long time (you might not have) and is helped by active marketing (you might not do).
This is more about quick-polishing your image than your ego. ;)
Researcher has to have some amount of authority in his field to receive grants. Publishing in high-tier venue is a good way to get that authority because it is how people-with-money judge researchers :-)
Well I really enjoy reading articles from top-tier venues (PL theory). It means that committee does a good work filtering and improving articles, definitely not without mistakes.
"What is the role of human scientists in an age when the frontiers of scientific inquiry have moved beyond the comprehension of humans?"
The above quote is from Ted Chiang's short story "The Evolution of Human Science," originally published in Nature as "Catching crumbs from the table" [0]. It's a brilliant depiction of this very problem: when new developments contribute to an increasing gap between those who can make new developments, and those attempting to understand the state of the art, the entire process of scientific inquiry becomes less efficient. In fact, the scenario depicted is one where the majority of researchers become "distillers," to use the language of the original post.
While Chiang posits a science-fiction reason for the divide, "normal" research/technical debt is insidious as well. Without incentives to reduce debt, the knowledge gap widens until only a handful of experts can make significant contributions. It's a problem that needs to be tackled head-on in both research and engineering. I'd love to see more initiatives like Distill.
There's another subtle aspect to this: the same or very similar ideas, methods, and tools show up or are reinvented again and again, under different guises, in different disciplines and subfields that have their own jargon, unnecessarily making human comprehension even harder.
If you look ahead to a future age, and consider the state of literature after the printing press, which never rests, has filled huge buildings with books, you will find again a twofold division of labor. Some will not do very much reading, but will instead devote themselves to investigations which will be new, or which they will believe to be new (for if we are even now ignorant of a part of what is contained in so many volumes published in all sorts of languages, they will know still less of what is contained in those same books, augmented as they will be by a hundred—a thousand—times as many more). The others, day laborers incapable of producing anything of their own, will be busy night and day leafing through these books, taking out of them fragments they consider worthy of being collected and preserved.
"If the government participates in such an opus, it will not get done. It should use its influence only to favor its execution. With a single word, a monarch can make a palace arise from the grass; but a society of men of letters is not like a herd of manual laborers. An encyclopedia cannot be ordered up. It is a labor that must be doggedly pursued rather than launched energetically... The more abstract the topic, the more we must strive to make it accessible to all readers... Such are the principal thoughts that came to my mind with regard to the project of a universal and analytical dictionary of human knowledge; its possibility, its purpose, its materials, the general and particular ordering of those materials, the style, method, references, nomenclature, the manuscript, the authors, the censors, the editors, and the typographer."
The translated full text [0] of that Diderot quote is pretty incredible. Two centuries prior to Wikipedia, he predicted many of the benefits and challenges of such an endeavor. I can only imagine his reaction if he were to see Wikipedia today; I imagine it would be something like [1].
Thanks for that. I'd only run across the fragment from which I'd quoted above, and even that is tremendously informed,sightful, and illuminating.
I've been digging into the history of information and communication (and only just scratched the surface of THoI website), including the explosion and dynamics of information as it spread across Europe.
There's a paper which looks into the total number of books (volumes in total, not simply titles), from about 600 AD through 1800. That grows from perhaps 30,000 volumes -- about 1 per million inhabitants -- to nearly 1 billion in 1800. The 15th century, with the invention of Gutenberg's press is an absolute watershed, and the consequences, social, religious, political, scientific, are immense.
I'm starting to wonder just what the hell it is we're unleashing with the Internet and in particular the Internet in your Pocket, across the world.
Not all the previous developments have been postive.
I can't count how many times I've invested meaningful time and effort to grok the key ideas and intuition of a new AI/DL/ML paper, only to feel that those ideas and intuitions could have been explained much better, less formally, with a couple of napkin diagrams.
Alas, authors normally have no incentive (or time, for that matter!) to publish nice concise explanations of their intuitions with easy-to-follow diagrams and clear notation... so the mountain of research debt continues to grow to the detriment of everyone.
I LOVE what Olah, Carter et al are trying to do here.
I really love this effort. Research papers are low bandwidth way to get information into our brains. They take a lot of effort to read, even if the ideas are not particularly complicated. Often when reading complicated material, I have to come up with metaphors in my mind to make sense of it. This is somewhat of a wasted effort, as the author who wrote the material surely had metaphors for their own mind when writing, but too often they don't share these metaphors, and stick to purely technical writing. I think this is one reason why ideas like general relativity are so popular, even though the material is actually quite complicated. The average educated person can give a reasonable explanation of general relativity because the metaphors used to explain it are so powerful, even though its very unlikely they understand any of the math involved.
I've often thought it would be great for the Arxiv to make it easy to link to a video of the 'talk' that generally goes with a paper. The talk is very often the distillation of the ideas in the paper, as conducted by one of the involved researchers. Indeed, one of the main points of conferences is to allow us to trade these distilled versions of our research with one another and place them in the context of everything else going on...
> I've often thought it would be great for the Arxiv to make it easy to link to a video of the 'talk' that generally goes with a paper.
This has been in the idea pipeline for a while, but the arXiv is slow-moving. (They probably wouldn't host the actual video files for cost reasons, but that's not a big deal.)
> The talk is very often the distillation of the ideas
This seemed a major omission of the OP.
The distillation quality hierarchy goes something like 'conversation among best-in-field researchers', 'conversation with principals' and 'research talk by principals, plus subsequent questions/conversations', 'review paper', 'journal commentary' and 'papers' but... in many fields, papers really need some of the above to provide critical context.
So a distillation system exists - it's just very very narrowly scoped. It's sad to see graduate students and young faculty who haven't "plugged-in", and so are not spending their time well.
And it's depressing to see a 'conversation among best-in-field researchers' excoriating a perspective on a topic, knowing that broken perspective is how the topic is pervasively taught, graduate down to high-school, and it's not going to change any year soon. There being few incentives or mechanism to pipeline such insight into educational content.
So yes, starting to build an expectation that talks become available as video, and are easily found, could be a non-trial step forward.
> Noise – Being a researcher is like standing in the middle of a construction site. Countless papers scream for your attention and there’s no easy way to filter or summarize them.2 We think noise is the main way experts experience research debt.
This is a big part of how I don't understand why some type of annotation standard hasn't taken off for research papers. Everyone does the same duplicative, time-consuming work of turning a paper into knowledge in their own head, so many wheels are reinvented. Where is the GitHub for research ideas?
I like the ideas put forth in this article. I wonder, though, if "distillation" is a re-casting of "scholarship" as considered by the humanities.
People studying topics ranging from Biblical Studies to History to Literature often do not create new source material, unlike in STEM. Yet there is a large degree of effort taken to "distill" existing facts through new lenses, producing novel concepts and interpretations. These efforts can transform our understanding of many areas of human endeavour.
One thing I believe to be of great value which is not made explicit in this article (which I think is an awesome article), is that research debt, as they describe it, is basically _education_.
In other words, improve the educational resources for complex subjects.
In our age where we're blessed with cheap printing of books and the possibility of creating complex interactive media, I think the question of designing user-friendly, powerful, and beautiful educational resources is a huge opportunity and pressing question.
Not just for people seeking to achieve a research-level understanding of a complex subject, but for all subjects and all people.
Consider the social value of beautiful, well-designed and nontrivial educational material for mathematics or basic science being widely available for all classes of people at all ages.
I'd argue that when news organizations use infographics or interactive journalism at its best, they are also performing this educational function.
Sorry for the long post, but to summarize, I think it's useful to recognize research debt as a specific case of the art and practice of creating media for education.
There are 'distillers' of large bodies of scientific research. Traditionally, they are science communicators, and more specifically science journalists.
The goal of a practicing scientist is very much at odds with someone whose job is to translate science into larger audiences. I've had very well-intentioned rational research scientists tell me with a straight face that "my job is to produce science results, not to communicate it. that's someone else's job", usually with the attitude that it's less respected or somehow self-aggrandizing. "The best science will be self-evident" attitude that all researchers secretly aspire for, not realizing that 99% of impactful science has had effort spent to promote, frame, or distribute it.
This weird stereotype is somehow beaten into scientists from the very beginning, and I haven't been able to figure out where this comes from. Obviously, yes, it's a lack of tools and accessibility into letting scientists also become distillers themselves. But the motivations and incentives at the center of the whole system is what's making this whole imbalance. I think there are parts of our research system that actually say "No, you cannot and should not distill your science".
Ultimately, for me, it gets back to funding. If review articles and outreach weighed just as much as citation count in tenure and grant committees, then maybe this could start to change. Yet, these committees still don't value open access, and look how tough that battle has been.
Also - this solution is really great and commendable, but I don't see how this works outside of ML/CS where research outputs are more like software development - gists, snippets, prototypes that are immediately shared, pushed, forked. More science fields like ecology, synthetic biology, anthropology, will look like this, but it will take a few human generations.
I think you miss the point. The article proposes a path for distillers whose target audience is researchers, not the lay public.
Science journalists are fairly poor distillers of knowledge, actually.
What's needed is more like a way for senior researchers to write more and better review papers that lay out and summarize all of the issues around a particular sub-field, for active researchers in that field.
Science communicators/journalists can turn "Methodological observation of the sociometrical behavior tendencies of prematurated isolates indicates that a casual relationship exists between groundward tropism and lachrimatory, or 'crying,' behavior forms." into "Scientists find that falling down makes babies cry" but they're less good at expressing concepts like "math equation forms the basis for all current modeling, papers A, B, and C each use a different special case of the equation to reach their conclusions."
Isaac Asimov anticipated the idea of a research distiller in "The Dead Past" with the character of Nimmo, a professional science writer:
"Nimmo received his first assignment at the age of twenty-five, after he had completed his apprenticeship and been out in the field for less than three months. It came in the shape of a clotted manuscript whose language would impart no glimmering of understanding to any reader, however qualified, without careful study and some inspired guesswork. Nimmo took it apart and put it together again (after five long and exasperating interviews with the authors, who were biophysicists), making the language taut and meaningful and smoothing the style to a pleasant gloss."
In the story Nimmo has less prestige than a "real researcher" but the role pays well and he is in high demand.
I don't see anything wrong with the "textbook" -> "review" -> "article" strategy to climbing a mountain of debt. A good review paper should hit a sweet spot in terms of exposition, digestion, abstraction, and noise filtering. Transformative new ways of visual thinking, well, that's a different story.
The problem is just that the pace of research in the authors field is too fast at the moment. What's the hurry? Over time, the citation graph will reveal the most significant work, and the community will naturally distill that research for maximum effect. The danger is that beautiful distillation of an extremely "niche topic" will not change the fact that it has limited scope and may even limit abstraction. Of course, I say danger with tounge-in-cheek...
You're right that reviews and textbooks do valuable distillation, but I'd claim we can do much much better.
I think we can distill things massively better -- create massively better explanations and ways of thinking about ideas -- if only we invested more in it. It isn't just when the distillation happens, it's the quality.
That's a really bold claim and all I can do is point to the occasional example where someone put effort into distilling and did something wonderful. My intuition is that these aren't one-off miracles, but are the expected result when the community takes this really seriously.
The problem is that distillation is requires focused energy. Reviews and textbooks give us a bit, but it's diffuse. I think we can do much better, and I hope we'll be able to demonstrate that with Distill.
As well as an underappreciated magnitude of expertise. You might like my "Scientific expertise is not broadly distributed - an underappreciated obstacle to creating better content" http://www.clarifyscience.info/part/MHjx6 .
> create massively better explanations and ways of thinking about ideas
Agreed.
My hobby project this month is: assuming it's possible to teach size down to atoms in early primary school [1], could we then leverage that to do a multi-scale interdisciplinary "nucleons-up" introduction to atoms and materials. Current content is so notably incoherent, and leaves both students and teachers so very steeped in misconceptions, that there seems an opportunity for supplemental content, if it can be shaped to work.
My experience points the other way. Surveys and textbooks are good in principle, but I rarely find myself using them in research. When you're looking to understand a very specific thing (e.g. what is ADMM? How fast does the conjugate gradient algorithm converge?), there's just far too much notational buy-in needed to get started in a survey, or god forbid to hunt down a textbook with a chapter on the subject. These are necessary evils if the topic is esoteric enough. But far more useful are lecture notes, powerpoint slides, technical articles, stackoverflow answers. These materials are scattered about the internet, and are useful even in their unpolished, typo-ridden state, is it any surprise they are the top hits on google? I am certain the community would benefit from polish and incentive to put these things down in paper.
Achieving a research-level understanding of a topic is quite different than working in the trenches on new research. I figured the goal of distill.pub was the former.
I think you're partially describing a search problem. Google fails to return relevant passages from full-text papers/books for whatever copyright reasons (ignoring the notational buy-in). The remaining issue is that reviews/surveys/textbooks do not provide the complete picture necessary to perform successful research. I'm sure they fall short on specific things that are scattered around the web; practicalities, implementation details, quick definitions, etc. But are you suggesting that distilled publications should include all this information in one place?
One counter-example of many from my field: Physics textbooks used at the elite institutions are awful. They have persisted for many decades, and there's no sign they will change.
I worry that these will be citations (and publications) that "don't really count" in heavily career-impacting decisions. Just getting citations to appear won't make funding agencies, tenure committees, etc. value distillation itself -- they will definitely know the difference between a survey paper and a novel result.
Yep. That's why a big part of Distill, behind the scenes, is a campaign of talking to senior faculty, scientific funders, and leaders of industry research groups, to build consensus that this is important.
I'm not sure how successful we'll be, but we're taking it seriously.
Scientists are supported and rewarded by the people who control the money in the ecosystem that they already live in.
I'm not sure that you'll be able to convince people to devote time and energy to a completely different ecosystem... one that their current ecosystem doesn't care about.
Narrowly construed, it is of course even beneficial to hoard knowledge; makes it more likely that your self/lab/friends are the ones that will publish the next advance in your field.
I do think that the ML / CS / etc. community is actually more open than other academic fields, and so this is definitely the right subfield to start in. Putting open access preprints online is not common practice in all disciplines, although it really should be.
I wonder if it makes sense for Distill to also publish on fields outside of pure ML - e.g. as applied to specific problems in other domains. I work in materials informatics, and I suspect that research in such fields (ML + applied sciences) might benefit quite a bit from having key results 'distilled' in this format.
In the very long run, I'd like to see Distill or Distill-like journals cover all of math/cs/science.
But I think the right approach is to start with a narrow topic and do really well there. I guess startup people would say that we're focusing on a single vertical. :)
We haven't done very good line drawing for journal scope yet, and I'm not sure how we'll handle cross-disciplinary work.
Article is really good, but it's reference to the pi vs tau debate is kind of silly. It really isn't a big deal. When I did my math phd I wrote 2 pi all the time, but this didn't matter. The tiny convenience gained by changing to tau is totally trivial and doesn't even deserve a mention compared to the rest of the things in the article.
2 pi doesn't seem like a big deal to you because you've internalized the notation. You paid the cost a long time ago (and perhaps it was a small one for you).
I think it can be hard to empathize with what it's like to be a beginner. I learned about the pi definition debate a few years after I learned trigonometry -- there were a bunch of essays prior to the tauday one -- and it seemed tremendously better. I was also still in high school at that point, and I saw students struggling daily with these ideas and it seemed like part of the pi definition was the stumbling block.
So, of course the pi vs 2 pi thing looks trivial to a mature mathematician: we aren't the people paying the cost.
That argument _may_ apply to very young learners. I.e. students in high school or younger. It certainly does _not_ apply to the people the original article is talking about (i.e. researchers). Anyone who can't handle the circumference of a circle being 2 pi has no chance at understanding modern mathematical research no matter how simplified it becomes. It's not even on the radar.
By the way I simply am not convinced by this 2 pi vs tau debate. Now the area of a unit circle would become tau / 2. The original problem doesn't go away it just changes. The argument can be that you write write tau more than you'd write tau / 2, but the fundamental difficulty remains. Notation in mathematics is _extremely_ important, but pi vs tau is so far down the rabbit hole of diminishing returns, that I just see the debate as a total waste of time (and that's before I consider the fact that it won't ever change due to momentum...).
The fact that you write down factors of 2 less is a weak signal of the improved notation, but it's not the point. (If you can't see that tau is conceptually a more fundamental quantity than pi, then you actually don't understand the point!)
The reason to use an elementary example like pi/tau is that it's accessible to a wide audience. Of course it's nearly trivial for experts. It wouldn't make sense for the author to pick a high-level stumbling block that's only important for a tiny group of experts in one subfield.
> (If you can't see that tau is conceptually a more fundamental quantity than pi, then you actually don't understand the point!)
Well then I guess I don't understand the point. That claim is essentially equivalent saying that circumference is more fundamental than area. You may accept such a strong claim, but I certainly do not.
It's not equivalent, but it's connected: circumference is "more fundamental" than area simply in the sense that 1 dimension is more fundamental than 2. In particular, when we build things up in higher dimensions, it's much more clearly done using a 1D primative than a 2D one, e.g., for the Fourier transform, there are powers of 2pi=Tau in any dimensions.
>> it's much more clearly done using a 1D primative than a 2D one
Or you could just tell people that pi is the edge length of half of a circle. Pi and Tau are both 1-dimensional numbers that happen to be related to these things, so the argument really is silly.
The reality is, we're going to have to pay some kind of intellectual debt in learning anything. So we keep teaching Pi. Suppose everyone switched to Tau. That would make all kinds of older documents inaccessible and harder to grasp, and we would be paying intellectual debt that way.
> Or you could just tell people that pi is the edge length of half of a circle.
You can certainly tell them that, and they may even find it easier to remember, but it won't actually be the reason that one convention is better than another.
> The reality is, we're going to have to pay some kind of...
Whether pi or Tau is preferable a priori is logically distinct from whether it makes sense to change conventions once one is established.
> You can certainly tell them that, and they may even find it easier to remember, but it won't actually be the reason that one convention is better than another.
Indeed. But it was arguing against the logic of the parent post, which argued that it was a reason for one convention being better than the other. Not to be rude, but please read the parent post before responding to a response. It tends to make more sense.
> Whether pi or Tau is preferable a priori is logically distinct from whether it makes sense to change conventions once one is established.
Agreed. But that's not what I was arguing. I was arguing that we have to pay some kind of intellectual debt regardless. I could have used a different example, such as cases where it makes more sense to use pi than tau, and in such cases, we would be paying some intellectual debt. But please, please try to see how the idea applies beyond the initial example. Don't just take the argument at face value.
I see tau as a thing that's mostly useful to programmers because writing "2*Math.PI" (or equivalents) everywhere is a good way to bloat up a trig-intensive algorithm. I have no comment on its applicability to other uses of mathematics.
Keeping the 2 around can be annoying in math for exactly the same reasons.
Also by the way, 2 * Math.PI is only one character less than Math.TAU. Even if you were to bind it locally to pi and tau, then 2 * pi is 4 characters whereas tau is 3. I mean it's a little annoying to keep track of, but it's certainly not the hardest thing programmers have to deal with.
edit: I'm a bit new to hacker news and am not really sure why I'm seeing these italics...
edit2: I see the issue is the asterix. Everywhere you see ' * ' imagine there are no spaces around them. I'm not sure how to "escape" an asterix...
If you're going to change notation there are things with much bigger payoff. I'd start by eliminating implicit multiplication. This does great harm to clarity by encouraging single character variable names.
You might like the Preface to the Structure and Interpretation of Classical Mechanics [1]: "Classical mechanics is deceptively simple. It is surprisingly easy to get the right answer with fallacious reasoning or without real understanding. Traditional mathematical notation contributes to this problem. Symbols have ambiguous meanings that depend on context, and often even change within a given context."
> I think it can be hard to empathize with what it's like to be a beginner.
In a similar vein, there's pi ≈ 3 (within 5% error). When we've an issue with high-school students having an infirm grasp of the very concept of volume, it seems an odd allocation of effort to have middle-school students struggle with 4/3 pi r^3, while failing to tell early primary-school students that "the volume and area of a ball are about half that of the box that wraps it".
> but it's reference to the pi vs tau debate is kind of silly
This struck me as odd as well, but here's a possible justification.
The ramshackleness of people's understanding, the degree to which it's compromised by misconceptions, grows surprisingly rapidly as one moves away from their specific research focus. A rule of thumb is a post-doc in a subfield, looks elsewhere in the field like a graduate or undergraduate student, and in other fields like an undergraduate or pre-college student. Because that's when they last wrestled with the subject. And science education research shows the wrestling isn't going very well.
It can be startling to be told by first-tier (non-astronomy) physical sciences grad students "the Sun doesn't have a color - it's lots of different colors - it's rainbow colored". Or to encounter first-tier medical school grad students, with no idea how big a red blood cell is, beyond "really really small".
With science and engineering education working so badly, I feel uncomfortable dismissing even "trivial" improvements to widespread abstractions, without looking for education research. I don't feel I can reliably judge how badly it might be screwing us.
When I've talked to senior researchers about these problems, they say that they have no problem finding distilled information about new results; they get it from in-person conversations at conferences that they attend frequently. The publishing of distilled research would most benefit low-status, newer researchers (like Ph.D. students), but it needs to be valued by senior researchers to make it into the incentive systems of hiring and grant funding. It seems like a tricky problem to fix the incentives here.
I think one of the biggest things missing is a means to communicate openly about published research. What I'd like to see is a forum for every paper, and this forum should be properly moderated (perhaps by a peer-review system).
Such a forum could make it much easier to decipher published work, and to fill in details which were missing. Also, errors in publications become clear more quickly.
Brilliant thing! As a person pursuing a PhD I'd something that is in my opinion the best way to avoid research debt: have a good tutor or a good group (peers). This way you can learn new things (or ask for something you don't understand) and really get to the peak sooner that doing everything alone.
3blue1brown's Essence of Linear Algebra series on YouTube should get some sort of honorary inclusion in Distill. There have been many "visualization of algorithms" posts on HN over the years. A collection of those would be good as well.
What I think is funny is that so many people complain about how bad science has become and how the papers and funding institutions enforce a very ineffective way of doing it. Why is nobody attempting or proposing new ways to make money with science? E.g. have people tried to do science in a subscription based model (like artists)? Or using free pappers+consulting on how to replicate the experiments or apply them in real projects (like open source)?
Ooooh. I've actually been working on a blog post where I try reading a science paper (knowing only a little about the science) and learn / explain as I go.
This seems like the better way to do it (my way is taking way to many words) - something like better science reporting.
My concern is the act of writing - Medium has a real nice web editor. What can I use to write an article using this HTML/CSS/JS without literally writing the HTML?
Shameless self promotion: I've been doing the same thing, for systemsy papers. http://blog.elvinyung.com/
I genuinely think it's really unfortunate that academic research are often so hard to approach without getting used to it. Often the concepts underneath are very cool and useful, and aren't actually that complicated. There is much to benefit from bridging the gap between academia and the wider community.
As an (aspirant) programming language theorist, we have a really great advantage in this field: advances in the field of prog lang have a natural distillation process: getting into a "real" programming language.
And it's great, it means I can toy with features and consider "what could I do if it was in a real language ?". Also, we can observe idioms and usage that gets developed when some advanced feature start being used by "normal" programmers.
Computer science, in general, does have the advantage that the distance to applications is quite often much shorter than math, which forces part of this distillation process to proceed a bit quicker.
On the other hand, the formalization is highly non-uniform, due to how young it is as a science.
In my experience this is already being practiced. Maybe it's different in machine learning, but in type theory simpler explanations are well worth publishing and are published all the time even in high impact venues.
As far as I can tell, progress towards eliminating "research debt" takes two forms: on the one hand there is a place for good exposition, typically in the form of textbooks, and on the other hand concepts get better understood over time and people come up with simpler explanations. In both cases the results can already be published...
Is the situation in machine learning really so bad that nobody is going to publish simpler explanations of known results?
How about we just abstract to "Intellectual Debt"? A subclass that I've recently encountered is "Learning Debt".
I think all these types of debt can equally apply to learning - as in formal education. Came to mind while reading "Undigested Ideas" because I recently had a conversation with my daughter (sophomore at Yale) that a bad test grade means that you may have Intellectual Debt that needs to be retired by taking time after the test to make sure you understand what you did wrong.
Here is a (well-publicized) post which only serves to hype up and stake a claim on what is already valued and practiced, the exposition of research. The message seeks to capture an ignorant readership and practitioners with short-term memory, and have them walk away with the thought "This is the place where good research expositions will be."
The machine learning learning community has already benefited from the myriad great expositions provided online for free, in addition to the locations where even source code is given alongside research findings. I will not link to these sites, in hopes to not seem a salesman, but if you've taken an interest in the field and taken some time to search on a topic (for example neural nets), you will have likely already found one of several free, helpful resources. This includes online courses, online books, blogs, videos, &c.
These individuals give examples of already well-written expositions going back years (associating these well-received expositions with their own endeavour), yet the theme is one of "newness", helped by the usage of terms like "distillers" and "research debt". In Silicon Valley, it seems that if something's been given enough press and sounds new, you should at least hop on the bandwagon for a while, lest you risk missing out on being an "early adopter".
Publishers are not a new conception either. They're a funnel which selects what you see. In an era where the larger population is beginning to see how much power publishers have over what they think, new efforts should strengthen decentralization. Playing to the tune of publishers has gotten us into the mess we're in.
Peer review is a mainstay of science, and I am in complete support of it, but peer review and publishers do not need to coexist. I and others will happily use a decentralized system with peer review, additionally and crucially providing transparency.
Good exposition should occur. Good exposition does occur. Indeed, people are learning: Ask yourself when you last learned from someone's writing. Audiences get their information from many sources, and if they can't understand those sources, they don't go to them.
Addendum. I will not discuss at length the repercussions of teaching the population how to create intelligent systems, but that is a dangerous road for all of us, and not one easily traversed. Companies and other powerful persons have a strong interest in guaranteeing they have a large pool of subordinates who have skills that they can profit greatly from, so it's obvious why the push for software engineering and artificial intelligence teaching is so strong (the promise and hype has been strong). But this is heavy-handed and short-sighted. Businesses have functioned with this approach in the past however, so I wager they assume past performance can be used for prediction in this case as well. If anyone's doing any thinking at all.
A normal person might use the term "teacher" or "science communicator" instead of "distiller". This is, however, from Silicon Valley, so naturally uninformed perspectives from individuals who don't seem to have spent substantial time deeply embedded in academia (adjuncting as a side job doesn't count) should be more valuable than insiders with actual expertise.
My scientific writing was transformed by Rafeal Luna's "Art of Scientific Storytelling". I'm on my phone, so I can't do it justice here, but I think that style will be crucial for the distillation TFA is discussing.
One take away I got from this is the immense value of great content marketing.
Especially for software, the best content marketing is an immense value to the community, helping it understand core ideas and adopt new technology more easily.
I agree 100% that it is way too hard to climb the mountains of knowledge, that it could be much easier, and that there is little incentive and cultural support in academia to work on these topics. Time and time again I see examples of where outcomes are not limited by the existing knowledge, but by the accessibility of said knowledge. I would gladly devote most of my time to distilling knowledge if I saw a viable career path for it.
Over the last years, I gave a lot of thought to the topic of conveying information. I see research as the process in which knowledge is created. This knowledge then has to be “encoded” in a format that allows for the transmission to other people. This encoding can be optimized in different ways. Research articles have the advantage that they are close to “lossless” in that they are supposed to contain all information necessary to build up that knowledge. This makes them well suited for archival, especially as they can be stored as a stack of paper.
However, research articles are often not optimized for building up that knowledge in an efficient way. I believe that the “encoding” optimized for learning & understanding should be more like a progressive image codec, in that it provides a comprehensive view as soon as possible, filling in further details along the way. This also makes it possible to stop whenever you have reached the level of detail that is relevant to you. The challenge in creating these encodings is to extract the information that provides the most clear & useful picture as soon as possible. I like the word “to distill” for that process, as it is really about extracting the essence of a body of information.
Doing this work for research articles is how I understand the goal of distill.pub, which seems extremely valuable to me. However, I think this is just the first step. What is the most useful distillation of all of deep neural networks? Of all of machine learning? Of all of computer science? As others mentioned, there are some forms of publications (review articles, textbooks) that do part of this distillation process, but they only cover part of the spectrum. Preciously few textbook contain a well thought-out summary of their contents and not just an introduction. In my experience, often the least amount of thought is given to the highest level of abstraction (e.g. what is the essence of mathematics?), even though they are the most fundamental ones.
It would be great if there was more focus on extracting useful understanding from the ocean of knowledge we already have (useful both in the sense of being applicable on its own as well as being a solid foundation to build more knowledge on). It looks like distill.pub is a step in that direction, and I really hope it will bring more attention and recognition to this kind of work.
Each Distill article is a github repository, so people can make pull requests, but we mostly encourage people to use that to make corrections.
If it's technically feasible to do community editing of articles, why don't we encourage that? It's actually a pretty deep question about how you think about research distillation.
It's tempting to see distillation as putting a layer of polish on ideas -- something that people can just jump into and do. But I think it's much more about deeply internally digesting ideas, perhaps with a few collaborators, and putting it together into a framework of thinking.
For that, you want a small number of people with a strong vision and a deep investment in the article. If other people have a different vision, they can write a different article.
(My impression is that wikipedia works very well for getting lots of facts onto a page, but does a much more mediocre job of distilling ideas.)
Wikipedia seems to be the closest thing we have to broad distillation right now.
It works pretty well if you are researching a subject in depth. The technical articles tend to be written by experts for experts and they have lots of links to primary sources.
But, as you hint at, they are poor quality because they are being written by a committee.
I would love to see a collection of community written distillation articles, as you describe. Perhaps with some kind of taxonomy like Wikipedia's.
Great! Do you mind expanding on why it doesn't work for you with emphasis on the relevant, objectifiable aspects? (as objectifiable as you can be for an anecdote, of course!)
What's actually being described in the article is entropy. Entropy counts always go up, and it takes energy to fight entropy. Or to put it another way, it takes more effort to produce simple writings, than it does to make complicated writings (especially for things that appear inherently complex).
This means that debt isn't the right term here, because debt is something that must be paid back by the individual, or organization that took out the loan. It also requires a lender, which, if we're talking about technical debt, is usually future you. In terms of publishing papers, the author isn't borrowing effort from his own future, he's simply not doing work that he doens't need to do. The problem the article is talking about is that others have to do work because the author didn't do work. This is not debt, it's something else (I'm sure there's a term for it in economics, the closest I can think of would be some subtle variant of a negative externality).
While I agree that research would be easier if authors spent more time distilling their ideas, that effort is a public good, and effort that will largely not have a positive expected ROI for the author, even if there is a huge ROI from a community collective point of view.
I agree that when you think about individual researchers it's a negative externality -- in many cases, the researcher doesn't pay, but other members of the research community do.
This is true in programming as well: often the technical debt falls on your collaborators. You understand that crazy code you wrote and didn't document, but no one else does.
For the debt perspective to really work, you need to think of the research community as a team working together to advance science. Individual members of that team take out debt to advance their work faster, but in the long run it costs the team more.
As a company or individual, when you take on debt, when you become "highly leveraged," all is good as long as that allows you to grow. But if your growth (and ability to repay) stagnates, then you enter a difficult situation where you are unable to effectively make (or attract) investment in themselves, because the overhead inherent in that debt load (interest payments, risk-averse behavior for fear of default) would eat away at the returns of any additional investment. That overhead must be paid down, at great distraction and opportunity cost, before future investment makes sense.
"Research debt" and "technical debt" are much the same; corners cut, such as lack of approachable documentation (which is exactly the "distillation" mentioned in the original post), may allow a small group of researchers/engineers to move quickly in the short term, but it can lead to development stagnation in the medium term as newcomers are unable to contribute; this is analogous to later investment being less effective. While many a CTO can understand the importance of paying down technical debt to allow for continued growth, it's much more difficult for a researcher to make that justification. So the analogy is a helpful one.
When programmers talk about technical debt, I understand the analogy. A project has technical debt if it has these early "corner cuts" that need to be paid off in order to move forward faster.
So wouldn't research debt be something a researcher has accumulated when she "cut corners" and didn't learn all she needed to learn? Who needs to pay off this debt?
And distillation as the opposite somehow of debt? Surely a researcher needs to pay off her own research debt by researching.
Sorry for nitpicking. It seems like people here enjoy the analogy.
Nobody's saying that researchers are cutting corners on their own learning; it's that once they make discoveries, they're not prioritizing effective communication to others in the field. When there's a general understanding that "someone's figured out a set of techniques for the state of the art in niche subfield X, but god help us if we want to actually build on those techniques, or evaluate their approach, because there's barely any learning path to get to that person's level" then research in that niche slows down, because new researchers are more incentivized to find areas where they can realistically push boundaries.
So it's not the researcher per say that accumulates debt, it's the research field overall.
It's exactly the same problem as if only a few experts in a software engineering group could be trusted to refactor a codebase because there aren't tests or other systems to help others get to their level of knowledge. And whereas engineering management might be able to tell the experts to disseminate their knowledge (pay down technical debt), in academia the experts will often just continue to work with the colleagues and collaborators already at their level. Pedagogy and prodigy don't always go hand in hand.
I tried (twice!) to publish a paper that was scientifically sound, but written such that it could be understood by a lay audience. It was rejected; quoting a reviewer: "This paper lacks math." That sentiment made me lose a lot of faith in academia. Now, I simply refuse to contribute any more time to writing papers or performing peer review (esp. in non-open publications). I'm sure I'm not alone; I know of at least one "seminal" robotics paper that was rejected from top venues multiple times for "simplicity" (lack of mathematics content) that went on to become a foundational paper in the field after appearing in a lower-tier venue years later.
The irony: it takes researchers a lot of time to make a paper dense & concise. If they "showed all steps", it would probably improve researcher productivity & make the material more approachable to newcomers. Instead publishers enforce length restrictions... for which authors dedicate upto 25% to related work (some of which is useful; much of which is pandering to likely peer reviewers in small, niche fields). Length restrictions seem equally foolish in the age of digital publishing. And again, inadvertent pedantry is the only explanation I can imagine... but happy to be wrong.