Thank you OP for your very insightful blog post. You have been better than any other academic at articulating exactly what happens when I (or one of my colleagues) interact with the academic world and you made me realize that the academic world has a set of values that might be good but which I should not even attempt to incorporate in my own professional life.
I have come to my personal conclusion that there is a difference between 'science' and 'the academic world' in much the same way as there is a difference between 'religion' and 'the church'. One can be very religious without going to the church and doing all the rites that are required by the church. In much the same way, a 'tinkerer' (or 'heathen') can be scientific without being academic. I believe that wiki summarizes it quite well for me: "Science is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe". There is no statement about giving proper credit (vanity) or not being allowed to reinvent the wheel without first making damn sure that you are indeed reinventing.
I want to solve real world problems and the way I want to do that is in a structural and repeatable approach. Science helps me doing this. Of course I'm not living in a vacuum, so I will look up as much as I can about the subject at hand as is reasonable within the available amount of time. Interestingly enough, I almost never end up with a research paper, but almost always with blog posts, books, tutorials et cetera. Products don't sell themselves and neither do research papers apparently. I think there is something seriously wrong if that is not perceived as a crisis in the academic world.
"a systematic enterprise that builds and organizes knowledge"
That is exactly my point. Doing your scholarship is doing your part to organize knowledge. Systematically building knowledge means building on others, and knowing what you build on.
So the quote (which you don't attribute to anyone other than "wiki" - a citation here would be useful) really proves my point. If you're doing not doing your scholarship, you are not doing science, or research, you are tinkering.
Interestingly enough, when I have found research papers on the subject that I am looking at, most are so opaque that it was not worth reading in the first place. More helpful information has been presented by others who you would classify as tinkerers. They have been clearer in their explanations and much easier to build upon.
In the sense of presenting the research in a manner that other people can digest, academia seems to be more of a ancient guild than an organisation for expanding knowledge.
Academic papers are written for an audience of experts. If you wish to understand them, you’ll need to be an expert in the field (you can sometimes get away with a bit less, depending on the topic). They are not intended for general dissemination.
>Academic papers are written for an audience of experts.
You and parent are saying the same thing. Just because they are written for experts does not mean they are not written in an opaque manner.
And as someone who was once in academia, they really are written for 2 reasons:
1. To get past the peer review process.
2. To be written in the minimal time possible.
Enlightening peers comes a distant 3rd.
Example: It took days for a grad student/professor to derive a formula that is included in the paper.
Professor insists the derivation not be included in the paper. Insists the student not even mention in a few sentences the steps to get to it. Claims "any expert should be able to do this. No need to add it to the paper."
It took that expert days to do it. Unless it turns out to be a seminal paper, I guarantee that in most cases, no reader of the paper will even try. An error in the derivation? No one would catch it. Clearly, the professor is not even writing for his peers.
I do agree with the parent - the practices approach that of a guild more than any objective measure of explaining things.
One could also argue its a lack of distillation. Yes, an expert in field could/should understand, but does that mean it should be accepted that the papers are inaccessible to without same or greater level of expertise? Even experts struggle with opaque nature of many papers. Not saying that they should be dumbed down or using less precise language in order to appeal to wider audience, but more than a little more effort could increase value to all.
Good point, I could have used "hacker" instead, though that's more specific to writing software. I didn't perceive "tinkerer" as a word with negative connotations when I wrote it.
Come to think of it, many outside the hacker community would perceive "hacker" more negatively than "tinkerer", because many people still think of hackers as criminal. At least that's my perception. I guess words have different values in different contexts.
> Science (from Latin scientia, meaning "knowledge")[2][3]:58 is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe.[a]
Incidentally, the "wiki" as a system separates "scholarship" from "authoring". "scholarship" is done collaboratively post factum, given substance "authored" by various overlapping sources.
> There is no statement about giving proper credit (vanity) or not being allowed to reinvent the wheel without first making damn sure that you are indeed reinventing.
Crediting someone is not about vanity, at all. It's so in the future someone else can see where to go for ideas you've used, what the general thought and progress was that lead to an idea. See "scholarship" in the linked article. Crediting people facilitates scholarship and knowing the context of an idea. In theory you should be able to trace all human thought that lead to an idea, and what else those people were thinking at the time. Amazingly in modern science that's essentially possible. Most papers have an acknowledgements section where you even acknowledge conversations or private letters that lead to an idea. It's not vanity, even in the world of tinkering its worth crediting people with the input for your ideas and the source of your material as it makes it easier for someone else to riff off of your tinkering.
It's not about avoiding reinventing the wheel either. It's about knowing all the criteria for wheelness, knowing what has been done before and what is widely accepted as a wheel rather than a roller or a track. Is there a body of related work that might apply in the abstract, if rollers are like wheels perhaps roller maths would be useful, perhaps a development in roller technology has not been applied in principal to wheels and could be a breakthrough.
Both of these together are how you actually push things forwards. Giving credit especially is highly misunderstood. If I want to know if one paper Im using is a weird off shoot of thought of the author or not, what else were they working on, where did their inspiration come from, is there more I can get from their direction, or is there somewhere I can go they never dreamed of etc. That transparency is actually amazingly powerful for the creative process.
> Products don't sell themselves and neither do research papers apparently. I think there is something seriously wrong if that is not perceived as a crisis in the academic world.
It's a case of expectation mismatch from your end. A scientific article is aimed to encapsulate all information which can help push the frontiers of human knowledge. Science is not meant to sell a product.
My point is that science is meant to 'sell' something. Not a product, but the knowledge that has been created. If the knowledge is never applied in the real world its missing out on a lot of potential. If 'tinkerers' are not able to digest the knowledge we do have an issue. I'd like to think that the tinkerers/hackers/programmers/makers in the end are the customers of computer science.
You are choosing to not engage with the author and simply repeating your own opinion. Its obvious to me that your disagreement is actually not even relevant to the article or to what the author is saying.
>There is no statement about giving proper credit (vanity) or not being allowed to reinvent the wheel without first making damn sure that you are indeed reinventing.
The article isn't a discussion about what science means. Its a discussion about the principles of research that all of our current scientific progress is based upon. One of those principles is the organization and structuring of knowledge. Citations are one of the means of linking knowledge that allows easy navigation between topics and ideas for people both inside and outside the field. Because of the collective effort of people citing other work, it saves time and money of everyone involved since they can easily hop between topics and ideas without having to read a million unorganized pages of scientific data and figure it out every single time.
> I think there is something seriously wrong if that is not perceived as a crisis in the academic world.
Um, you don't really get to tell someone they're doing it wrong, when what they're doing has worked out brilliantly for the human race.
I might not have been clear enough about the point I was trying to make.
OP's comment on the Super Mario Bros neural network was really insightful to me: "Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship". This made me understand his, and most likely the academic, point of view. What I tried to convey is that I believe that there is value in embedding your knowledge in the network of previous knowledge but the academic world has perhaps gone too far in focusing on this aspect and too little on making sure that the created knowledge is applied.
In regard to saying that something is seriously wrong; There is no doubt that the academic world has produced a lot of valuable knowledge during her existence. It's the current academic world that is in crisis. With the H-index as its ultimate false god. The H-index exists only since 2005. Why should I not be allowed to criticize that? If the system is still good, it will survive the criticism.
>but the academic world has perhaps gone too far in focusing on this aspect and too little on making sure that the created knowledge is applied.
Okay, now I understand the point. But why is it necessarily the responsibility of a researcher to find industry applications? The industry has to figure out which portions of these ideas can be realistically turned into applications or products based on market criteria. Of course industry can and often does contribute papers to further our knowledge. And when they do invest in doing research they're probably going to be motivated to find applications.
> It's the current academic world that is in crisis. With the H-index as its ultimate false god. The H-index exists only since 2005. Why should I not be allowed to criticize that? If the system is still good, it will survive the criticism.
I was using 'don't get to' as a rhetorical device. In my view it was like walking up to Usain Bolt and telling him how to fix the problems with his form cause hes doing it all wrong. Sure, you can do that, but most people won't take you seriously. Or in other words, simply being able to identify surface level problems, doesn't mean much without a ton of analysis, explanations, etc, etc. But yeah, obviously you can opine on any subject you like, and people too have the freedom to respond.
> I almost never end up with a research paper, but almost always with blog posts, books, tutorials et cetera.
That is probably related to your field of work. I once had to write a pathfinding algorithm for multiple robots in a tight space. With such problems, you will end up with research papers. I can imagine lots of other technologies (such as everything related to AI) where papers will help you the most.
Blog posts etc are a solution to common problems. Sometimes you indeed find some gems on architectural level, or for game development there's lots of valuable content out there.
But once you go to complex algorithms, that's the space when research becomes handy.
Similarly, a few years ago I swas developing a path-finding algorithm for time-varying 3D vector fields and I found myself knee-deep in various research papers. Some were better than others, and I did find a few web/blog resources helpful, but it was mainly research papers.
Research papers are typically about very specific topics. It would be strange to end up reading one on the Internet unless you already know the field keywords and use google scholar or any other science-oriented browser.
Indeed, google scholar and similar citation graphs are amazing. It makes tracking down vaguely relevant papers much easier than in the past. I can skim a few dozen papers one day, and then more closely inspect the handful that seem most relevant or interesting.
I wish that google scholar showed bidirectional links though. I suspect they might have concluded that the list of citations included in a particular paper is part of the content of the paper (not just metadata) so that showing it might infringe copyright.
I'm curious that you're taking issue specifically with taking credit and reinventing the wheel. Avoiding those things is the same as not stealing. I don't think it's academic at all. If you're using someone else's idea, and you just don't want to put some effort into finding the original owner of that idea, don't blame the author for it.
I thought you were going to go the other way on that one... going to church doesn't make you religious. I might even say not all churches have religion. And the analogy still works, 'the academic world' is hardly a subset of 'science'
I'm an academic researcher. I disagree with most of the article. The main distinction I see between "research" and "tinkering" is not what the author calls "scholarship". From personal experience the only serious distinction is that "researchers" tend to have a very long time attention span compared with "tinkerers".
As for the rest, the goal of academic research, lately, seems to be to produce as many as possible units for the benefit of the publishing industry. In the case of very successful articles, you can fit all the readers in a (small) bus. Most of the articles are badly written, because they follow old habits from the age when printing was expensive. Citations are done in a very selective way. Not so long ago there was no need to give the source of an "unpublished" article, "preprint" was enough. Tinkerers work is almost never cited, even if it is read and used. Blogs? are you kidding me? that's not "scholarship". Heck, even arXiv still seems exotic in some fields, despite the fact that it is a good, standard communication mean for the researchers in other fields. Finally, peer review. How does it work? Researcher B reads what researcher A reports and expresses an opinion, just like any honest reader could.This passes for validation. Magic.
In conclusion I believe the future of academic research comes from "tinkering" with a long time attention span. The dissemination is technically trivial. Peer review will be supplemented by more rigorous validation (although there is no absolute solution here). The same kind of validation will be applied to tinkerers results.
> The main distinction I see between "research" and "tinkering" is not what the author calls "scholarship". From personal experience the only serious distinction is that "researchers" tend to have a very long time attention span compared with "tinkerers".
Many aspects of "scholarship" become inevitable when "tinkering over a long timeframe". You build up a knowledge base about the problem and/or methodology you're tinkering with, you eventually form a community around your tinkering, etc.
Lots and lots of tinkerers gratify themselves without contributing any modicum of knowledge to the world. But the most successful tinkerers are almost always effectively scholars in the sense of this post. In the sense that they're familiar with a body of knowledge/the people producing that knowledge. And they contribute genuinely new ways of doing things, or new ideas of things to do, or excellent executions on existing ideas that achieve the aspirations of previously articulated ideas. E.g., Linus is certainly a successful OS scholar in the sense of this post.
> In the case of very successful articles, you can fit all the readers in a (small) bus
This is off by at least an order of magnitude for any even remotely reasonable definition of "successful".
> Researcher B reads what researcher A reports and expresses an opinion, just like any honest reader could
First, the author explicitly kinda-sorta agrees with you here (...Third, it's not really the publication...), so it's pretty clear that publication venue != scholarship.
As an aside, peer review != validation!!! This is something that every young Ph.D. should explicitly learn early on.
Yes, I hope that evolution will produce something new from the available academic and tinkerer genes. The academic lives longer in a stable medium, the tinkerer dies young but multiplies faster in chaotic conditions.
There are also some unfortunate features of academic research. A big one is that the product the researcher cares about is papers and citations, not working software per se. Papers you only write once. You can cherry-pick examples, work around bugs, do some facile analysis, publish the paper and then move on never to look at the code again.
This means that code and data is typically hacky, unpolished, poorly documented, unmaintained, and unsuitable for public use (or often not publicly inspectable at all). Often the folks doing the grunt work are graduate students, who leave after a few years, leaving a new batch of graduate students to deal with the code without much guidance, so it tends to accrete features but seldom get any refactoring love. Nobody is paying for bug fixes or software maintenance or documentation.
Someone coming from industry (or even just a tinkerer) who wants to use the same idea in a production environment often realizes halfway through reimplementation that it isn’t actually better than existing alternatives, and that the analysis/benchmarks in the paper were limited or misleading for a general context. Sorting the wheat from the chaff can be pretty demanding, ideas can sometimes be obfuscated by abstruse mathematical formalism, and academic researchers don’t typically go out of their way to make their output usable for non-academics.
Indeed, Tinkerer vs Researcher is a false dichotomy.
Far Better would be to compare Maker vs Researcher.
How "Production Ready" is the result of both activities is the interesting question. How well covered with tests? How maintainable? How re-usable in broader contexts? How readable and understandable?
I used to live on the margins of the Physics community and blamed myself for being a really really Bad Researcher for having no patience with the Physics papers I read.
Then one day I had an epiphany.
If I were to look at the physics journals the same way as I look documentation for software or hardware.... I would be really really Angry.
The average research paper is extraordinarily badly written and unreadable. They have adopted as best practices, styles and habits, that you'd never forgive in a datasheet.
If you picked up a datasheet written like a physics paper....
You'd throw it against the wall and spec a different part from a semi-decent manufacturer.
(Hint: Datasheets are notorious for being terse and indigestible and badly written.)
If you are not from a first world university with a first world library at hand with paid for subscriptions to the premium journals and a fat budget to cover pagefees... Scholarship be damned. It's a rort. It's a racket supporting a closed shop guild.
It's not Good Manners except in the sense of Manners defining a "Gentlemens Club".
A few years ago I found myself reading pretty much every paper on methods that could give a better looking solution to "deblurring" an image than the more established methods. This is really a field which has overlap with many different areas of research.
You've got traditional iterative methods through to bayesian methods and machine learning methods. Each of these approaches seemed to be relatively isolated from each other and largely cited from their own subfield, despite all solving variants of the same problem. When they did cite outside the field, they tended to cite less relevant or far from state-of-the-art papers.
What was amazing was the number of papers where you would have a plot showing how amazing the new deconvolution method was in the presence of noise compared to other methods, often using a vaguely defined metric. This plot would take up about half a page, usually following a few example images that were set very very small, often ~2cm and usually in a grid. That was always deeply suspicious.
The other thing I remember were models with tons of unexplained parameters, which were carefully tuned to give good results for that image. But really it was guesswork. Sometimes the authors would carefully avoid well-cited work in another field that they could easily have found.
As a tinkerer with a scarcity of time you learn where not to waste your time and move on.
Having tried to find image-processing research/scholarship around 2009, before the neural net explosion really set-in, I have to say the field was a complete rag-bag of fragile, vague methods. And efforts to extend these didn't help them.
Which is to say that no all incremental, scholarly research is necessarily what's needed. In fact, I think what the OP describes is effectively what Thomas Kuhn calls "normal science" [1], which can be great until it becomes terrible - ie, "normal science" periodically reaches crises that require a different approach (inspiration, insight, iconoclasm, etc, whatever).
But that shouldn't detract from the point that anyone should know what traditional research, that if someone is going to engage in something other than normal science, they would benefit from being really aware of why and how they are doing this.
> a few example images that were set very very small, often ~2cm and usually in a grid
Journals sometimes require the authors to pay "page charges" of sometimes hundreds of dollars per page. I know of some (relatively high impact) journals in image processing related fields that require this. If the author is not working under a grant, then there's not much incentive to add to the page length.
Of course, this is insane that this is even a thing in the digital era, and it feels very mobbish. Idk. Maybe there's valid reason that I'm not aware.
It’s funny how some disciplines banded together decades ago to found their own open access journals, but other disciplines are still train wrecks in that regard.
As a researcher, I'm agree with your point. One important aspect of science/scholarship is reproduciblity [1]. There is a crisis in current science regarding this. The response to this crisis is the open science movement. Some protocols on how this body of knowledge is maintained should be change but this takes time.
Well, the thing with academic research is that it involves a "community of researcher" whose activities involve "evolving an idea". And in such a situation, the result is primarily expressed by its text/human-language representation while code is secondary to this. While this is problematic to verifying the results, it should be noted that code as a result has significant problems. Code doesn't have a long history of being read, interrogated and modified while human-language does. As the article effectively says, don't underestimate the significant value of putting a result in historical context. A human-language description can give a clue whether X has been done before, perfectly functioning code might not.
This question touches on "programming literacy", which in the extreme asks, "how can computer programs also search as _documents_ in the fashion scientific papers have historically served as documents." So far, I don't think any answer has really appeared. Perhaps that problem itself is "AI complete", present either a terrible or green pastures for research.
True, but sometimes you commercialize something with some insights you bring from industry that never would have occurred to the original academic that made the piece in the first place. Basically what I and another guy did in 2012. We were stupid though, took the first real acquisition offer. That's the problem with being in your twenties you want money ASAP.
"$1m today or $10m three years from now? Take the $1m, we'll figure out how to get the other millions later."
I think it's another reason why the rich keep getting richer. Muuuuuch easier to turn down $1m when daddy is worth $50m.
Sometimes the alternative to published research is no material at all.
In this instance you need to pay the price of working in an obscure domain, research literature, and experiment what works and how.
As a concrete example, boolean operations on polygon meshes still have even not one "de facto" implementation. You can find papers from early 70:s, but each method has one catch or another. Compare this to, for example to sort alogrithms, where just quoting the method name gives sufficient information - "quicksort or mergesort"?
The only source are few obscure books,a ton of research papers, and some open source libraries with very little documentation on the algorithmic specifics.
In this instance I'm very happy the research papers exist - they give lots of information and ideas, on what has been tried and so on. If not for the papers, the occasional tinkerer would have all of that ... decades of academic research ... to fumble through, all alone.
At least some of academia recognise and acknowledge the state of their "hacky, unpolished, poorly documented, unmaintained, and unsuitable for public use" code:
> ideas can sometimes be obfuscated by abstruse mathematical formalism
They are trying to communicate with other researchers who actually care that the proofs are correct. Maybe you don't, if you're simply trying to reimplement some algorithm, but criticizing research papers for using mathematical formalism is a bit ridiculous.
One thing I'd note is that before that Internet era, the scholarship needed to discover who had done X previously took a really significant amount of time. The Internet has changed that a lot, at the same that it has tempted people into tinkering and overall wingnuttery. I remember in 80s, having professor who functioned as little more than a search-engine, tell him an idea and, after a couple of puffs of his pipe, he'd give you a list of people who'd worked on it. He was marvelous despite apparently never having done much more than providing this kind of information.
Another thing to note that scholarship implies a "community of researchers". If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything. The article kind of says this but also consider that the people who have worked on this stuff exist, they have publications, conferences, meetups, etc and you can go to these or look at these and find-out whether or not your idea was considered and rejected, think about what's different between your idea and the idea the "field" rejected and so-forth.
> If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything.
At the same time, such reasoning is used to dismiss or discourage new attempts, often unfairly:
- each of the earlier people were approaching that problem from a specific angle, and new people might be approaching it from an untried angle
- the earlier people might have had limited time to really investigate the matter, so there may be unturned stones
- advances since the earlier work may give newer researchers a leg up.
Note that all of these apply even if (as is likely) the earlier researchers were very smart.
>>If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything.
> At the same time, such reasoning is used to dismiss or discourage new attempts, often unfairly
Well, if you use such an approach of considering previous work, you will have the tools to show you why your work shouldn't be dismissed (whether you are believed or not is a different matter). If you aren't looking at earlier work, not only do people have good ammunition for dismissing your position but it really is hard to be sure you haven't just repeated something.
> if you use such an approach of considering previous work, you will have the tools to show you why your work shouldn't be dismissed (whether you are believed or not is a different matter)
That doesn't mean you'll be listened to. You might, but that's no guarantee.
There's also a situation you're overlooking: people being discouraged for the cited reason before they even start. "What's the point, if people as smart as you -- and possibly smarter, more senior, and with more of a reputation -- already tried to think of everything?". This can be something that the researcher can tell themselves.
I was explaining why there can still be room for progress, even if lots of smart people have already explored the problem.
> If you aren't looking at earlier work
At no point was I suggesting people shouldn't look at earlier work.
Well, as Thomas Kuhn said, science tends to proceed incrementally under normal circumstances. When science reaches a crisis, then there is a tendency to look towards solutions outside the normal paradigms.
Conveniently, computer programming and invention offer other ways for what the OP called "tinkers" to demonstrate the effectiveness of their ideas. Science is imperfect, especially isn't necessarily about the latest, most exciting ideas but about making ideas reliable and the problem with being willing to quickly accept even plausible-sounding ideas is that they can also include a lot of dreck. But hey, there are many alternatives to scientific acceptance.
I'm familiar with Kuhn. I'm not saying we should be quick to accept anything, nor am I making any point related to science vs tinkering. All I'm saying is that there are two sides to the statement I was originally responding to, and that they should both be considered.
> One thing I'd note is that before that Internet era, the scholarship needed to discover who had done X previously took a really significant amount of time. The Internet has changed that a lot
It's exactly the opposite. You can now spend years just reading all the recent papers relevant to a quite specific topic of interest, and 20-80% of your time keeping up with new results.
If you narrow your research so much it's manageable you're immediately rejecting lots of directions right from the start.
> First of all, it's not about whether you work at a university and have a PhD. People can do excellent research without a PhD, and certainly not everything that a PhD-holder does deserves to called research.
I appreciate this dismissed as a straw man, but there are definitely online communities (and individuals I know) who reflexively discount comments from people outside academia.
I understand the instinct, it's a reasonable immune response when everyone thinks they're an expert because they read a blog post or a pop sci best seller.
Sometimes, though, the antibodies kill perfectly healthy tissue too, like valid contributions from bright generalists.
another factor is that you are very defensive if you spent a bunch of time getting a phd and someone didnt and is still able to do research as well as you. and im sure its not very common, since getting good at something requires lots of time and energy. but i bet it does sometimes, and thats probably one of the reasons why some of the healthy tissue is killed
And they have families to feed and their income depends on thier PhD being enough of a barrier to entry that they can pay thier bills. So much human progress has been lost because the economic model is broken. We seemed to make more progress when science was restricted to a smaller group of practitioners, many of whom were wealthy enough that they could be guided by the science rather than the funding.
Also in the early days there was more low-hanging fruit. You could make scientific discoveries by locking yourself in a room for a week and thinking deeply about something you read recently. Then come out and test it with items from your kitchen.
Now it's harder to make breakthroughs without specialized equipment, and the area of interest has narrowed to a tiny sliver.
Funny thing: do your eyes light up seeing a "scholarly academic paper", or a "deeply tinkering blog"? Which one have we trained our brains to react more strongly to, what is more exciting to you? What do you feel you learn more out of, have more confidence to choose as a starting point for further activities?
OP's distinction between tinkering and research seems wishful thinking at best.
There are many grievances people have with research (and it's easy to think of examples), but I find most tend to fall under:
* (unaddressed) RISK: poor result analysis, cherry-picked examples; glossing over edge cases and failure modes; focus on novelty and "academic writing" at the cost of repeatability and conceptual clarity.
The Mummy Effect of research that looks great but crumbles to dust when you actually try to touch it: https://rare-technologies.com/mummy-effect-bridging-gap-betw...
* (missing) OWNERSHIP: cadence dictated by grant cycles rather than one's intrinsic motivation to solve a problem; slow or missing feedback cycles; no dog-fooding => poor code and documentation; few incentives to make it easier for others to pick up or validate the results
In fact, both of these seem better addressed by tinkerers than by institutionalized research. No wonder people react more positively to "tinkerers" and "hackers"! Academia has a lot of positive inertia (and taxes) going for it, but all goodwill has its limits.
As a self-defined tinkerer, I have tried the transition to a peer-review recognised researcher. It is very hard. Science is a very harsh mistress, and what you want your work to be saying is quite distinct from strong tests into what it actually says.
The game-like qualities of surviving peer-review are just that: a set of pretty arbitrary hurdles to cross. If you work in a narrow enough field, your set of blind review peers is tiny and their critique, even if masked, can seem very personal at times. I don't like that much.
I don't overall judge my attempt a success. I think this is an endevour best started early. Late stage career, trying to demonstrate you understand the rules of science, is hard.
The best way to get your work accepted in a peer reviewed venue (journal, conference, etc) is to voraciously consume everything that venue has published before, and make your work match the style observed + copiously quote works from that venue.
Obviously the intrinsic quality of your work matters. But you’ll more often see work of marginal value that properly follows the form than work of great value that does not follow the form. In fact, in my field (academic HCI), I have never seen the latter, yet see plenty of the former at every conference I attend.
This is more important than you'd initially think. Incentives matter, and professors are rewarded for publishing papers that get cited by other papers. So if you cite your reviewer's paper, it's in your reviewer's best interest to get your paper through review. Hence, quote a ton of works from that venue and hope to get lucky like that.
Replying here because this matched my train of thought well.
I'm actually really curious to know if anyone has first-hand knowledge of a tinkerer (by this blog's definition) getting their work published.
As the GP said, understanding the rules is one thing, but demonstrating their understanding is another. And as you said, matching the style seems to be more important than the value of work that is demonstrated.
I see the value in matching terms, jargon, style etc just so the reviewers can standardize their thought processes. But as someone who's been a tinkerer for ages now it's hard to change styles for no immediate benefit.
So my first attempt was 'this is what we do' -a descriptive paper, documenting a technique. The reviews that came back were basically a mixture of 'so what' and 'what does it show' and 'show something of significance' -I had tried to write it as the ur-paper, to establish a technique, and only described in general terms its applicability. They wanted a lot more demonstrable outcomes. I walked away. I found peer review very upsetting. It felt like nobody actually cared about what I was trying to say.
My second attempt was 'this is a specific thing it can do' combined with a much more rigorous academic 'this is the analytical technique' and 'this is a polemic about lack of statistical rigour in results, here is our data, you repeat it' -Which interestingly got panned as 'too much tutorial, too much argument, more results' -which of course this time, we addressed instead of walking away. Result? we didn't get in the first journal/conference, we made the second. I'm reasonably content, but this feels like 'learn the rules of the game' a lot more than 'say something of merit you find personally interesting'
Oh, and 'this technique is interesting' doesn't seem to cut it as a paper subject.
Sorry if this comes across as harsh but I'm really not understanding your grievance.
> They wanted a lot more demonstrable outcomes. I walked away. I found peer review very upsetting. It felt like nobody actually cared about what I was trying to say.
> Oh, and 'this technique is interesting' doesn't seem to cut it as a paper subject.
Well, yeah? To invoke an HN cliche, "ideas are cheap". Why should anyone else care about your idea if you can't be bothered to show it actually does something interesting on some specific problem(s) or even motivate why it might be expected to do something interesting in light of what's already out there? Without any expectation of experimental validation, conferences would basically be giant circle-jerks filled with completely inconsequential "interesting ideas".
And it sounds like you took the feedback from your first round of peer review and revised your work in light of those critiques and got your resubmission accepted. That seems like a pretty good experience to me, knowing many academics with multiple experiences of resubmitting work 3+ times (with new results and revisions each iteration) before acceptance. I'm not saying that any peer review process is perfect by any means, but it's a very important filter and in this case it honestly sounds like the criticism you got when your paper rejected was pretty fair...
Assuming you're talking about academic computer science papers, you're right that it is not "say something of merit you find personally interesting". It is more "convince the reader that something you find personally interesting has merit."
I like this differentiation. Early on in the 3d printer craze I would often say "A lot of people are building 3D printers but only a few people are engineering them." which was much the same argument. When you "engineer" a thing you do so understanding the constraints and design it to reliably within those constraints. When you just build something, as soon as it works once you're done.
This reminds me of the quote from one of my favorite shows, MythBusters. “Remember kids, the only difference between Science and screwing around is writing it down.”
One is a crude and improvised or temporary solution to a problem, designed to be more functional and timely than precise. Something done with little forethought, organization, planning or precision.
Whereas the other is meticulously documented to note the choreographed nuances.
agreed as long as you also append 'to specifications under all acceptable conditions.'
It was the latter bit that really challenged 3D printer makers. So many 3D printers would work one time and not the next because there were so many parameters that they didn't specifically account for or respond to.
As someone who spends a fair amount of time with software developed by researchers (mostly biochemistry visualization software), that for all the "research into what has been done", I think there is a huge amount of effort being duplicated all over the place. In my experience this is due to the relatively short timelines of grants, the fact that you have to spend most of your grant money demonstrating that what you created has some kind of scientific impact vs actually implementing the thing, and, as others have pointed out, the fact that a lot of the core work is done by diamond-in-the-rough superstar grad students who will move on after 2-4 years.
I wonder if the author would classify the Wright brothers as 'tinkerers'. Overall I find the tone pretty insulting, nobody believes they are the bees-knees quite like academics.
Some time ago I wrote about academia vs industry in the context of data science:
In academia, you are allowed to cherry-pick an artificial problem and work on it for 2 years. The result needs to be novel, and you need to research previous and similar solutions. The solution needs to be perfect, even if not on time.
In industry, you should solve a given problem end-to-end. Things need to work, and there is little difference if it is based on an academic paper, usage of an existing library, your own code or an impromptu hack. The solution needs to be on time, even if just good enough and based on shady and poorly understood assumptions.
So, contrary to its name, data science is rarely science. That is, in data science the emphasis is on practical results (like in engineering) - not proofs, mathematical purity or rigour characteristic to academic science.
> So, contrary to its name, data science is rarely science. That is, in data science the emphasis is on practical results (like in engineering) - not proofs, mathematical purity or rigour characteristic to academic science.
Science in a nutshell is to take a hypothesis, test it rigorously and validate/invalidate the hypothesis based on the results. The results of this experiment could be extremely practical. Data science that follows the scientific experimental design, does science. The one that doesn't is equivalent to reproduction of past experimental results OR putting a blindfold on your eyes and throwing darts around until it hits the bull's eye.
I disagree with this common, yet faulty, explanation. (IMHO coming from people who have never worked in science and for whom everything R&D looks like "science!".)
Building engines or cars (involving a lot of tests!) is considered engineering. And as any serious engineering it's not purely random trails&errors. A/B tests are not a sufficient criterion to make something science. And data science (in business) is focused on solving practical problems (which may or may not be research) rather than open-ended research (which may or may not be practical).
Source: I worked in academia. I now work in data science. You?
I have a Ph.D in Computer Science and I have worked across research labs in the industry too so I very much understand where you come from. That said, I don't see the disagreement between us and what part of my explanation you found faulty.
I'm the author, and I'm somewhat amused by the negative tone of many of the comments. Seems there are plenty of people with resentments towards research here ;)
I think it's because it treats hackers as deficient academics who can't be bothered to cite their sources properly or do any research, while there's an equal and opposite argument that academics are deficient hackers who can't be bothered to put their code on github or write any documentation.
In reality, neither is really deficient, they're just aiming at different audiences which need different things.
Oh yes, I'm a terrible hacker, I rarely write code anymore. An unfortunate byproduct of academia is that you get "promoted out of the job", and all the hands-on work is done by your lab members.
I didn't mean to imply that tinkering was inferior to research - the whole premise was just to tease out how they're different, with different audiences, as you say. Interestingly, the discussion here has been dominated by people who think that I look down on them. People who've discussed it in other fora have not read the post that way.
which cites two [1][2] of your papers among others :
yet you say :
"
it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up.
"
a bit harsh perhaps ? At least mentioning the foundational NEAT paper is not a complete lack of scholarship.
I feel you have overlooked that Seth does try and provide good citations to his audience and it would kind if you mentioned him by name in your article rather than " some guy "
He did inspire other twitch streamers to experiment with neuroevolution and neural nets and benchmark many SNES games.[3] although downstream this follow up does goes uncited by yourself.
My main criticism with your otherwise excellent article is the use of the word "tinkerer". "Tinker" has a pejorative meaning: "to busy oneself with a thing without useful results", "to work unskillfully or clumsily at anything." to take a couple of definitions from a random dictionary search.
I'm old enough to remember "shared source" and the classification of "hobbyists" in Microsoft's licenses. In that case the intent was to try to create a division between "professional" programmers, who would pay for useful access to source code and "hobbyists", who supposedly had no need for useful access to source code (because they aren't "professional").
I'm quite certain that's not your intent (and you say so several times in your article), but the implication still lingers. My area of interest is in language design and practitioners have made significant contributions to the field. Researchers are enabled because the endless experimentation of practitioners have narrowed the search space, even though it may have been done in an inefficient manner (by reinventing the wheel many more times than necessary). In turn practitioners have benefited greatly from the exhaustive knowledge and documentation of the researchers.
Both scholarship and practice are useful, but it is understandable that one group may emphasise one over the other and get different kinds of useful results.
I don't think you will find it against research per se, but against the arrogance that many academic researchers exhibit towards those who are not in academia. Once one has a PhD, it seems to be a mental requirement that one will have or show disdain to those who do not.
I have met humble academics, I have met many more who are arrogant know-it-alls. It behooves all to understand that each one of us has a limited set of knowledge. Especially when one becomes more "expert" in some field.
I think one reason that various groups of people has a distaste for science and academics is the fairly common "I'm better than you because I've earned a degree or two or three and know more than you do" attitude displayed.
One aspect of research and publication which has been highlighted many times is that replication of experiments that have been published don't get any funding or recognition.
In that regard, what seems to happen in research is that instead of further investigating the slight anomalies that are found, it is often just let go because it is not within the scope of what was initially being investigated. It seems to be a rendition of the attitude of "Move along, there's nothing to see here".
In case it was unclear, I personally am not at all resentful toward academic research. It is just that for structural reasons related to social organization and funding model, it has both advantages and disadvantages, and the latter shouldn’t be ignored.
Your post did a nice job explaining some of the reasons that academics do the type of work they do, and how that differs from hobby side projects. I think it’s more interesting to discuss the trade-offs involved instead of just passively agreeing with the OP. [Also, many non-academic projects are more organized than someone’s weekend hobby hack (large scale volunteer efforts, funded by donations and sponsorships, directly corporate-run, etc.), so it is worth discussing those as well.]
My comment is probably also considered "negative" but I actually come from an academic background and appreciate the value and importance of good research.
In my opinion, the major problem with your post is that your view crystallises the ivory tower mentality instead of really discussing the difference between "researchers" and "tinkerers". In reality, statuses and titles aside, there is only a fine line between a seriously good "tinkerer" and a seriously good "researcher", and the assertions you have made are simply shallow and narrow.
Your attitude towards opposing, but non-hostile opinions is also rather off-putting—you are simply making light of people who wish to add to the discussion and dismissing them instead of addressing them. It doesn't seem to be that of a reasonable person who is willing to consider different views and explore different possibilities, and definitely not one of a good researcher.
It's hard to have a conversation when so many have angst or animosity toward academia. how many times have I read denigrating opinions of the validity of research here on HN
I suspect that some people here have had to take software written by academics and convert it to production quality code. From experience it's a pretty thankless task :)
Whenever i read a paper, i do have troubles understanding the fine details.
That one math formular which is nowhere explained? Yeah good...
I often enough, get the idea and details like results but thats it.
Those papers are not tutorials and they are not written / done to be reproduced and easy understandable. They are, in my opinion, written for other science people who spend there work time doing something similiar or need to solve the problem what the paper solves and rebuild those results in time consuming work.
Probably still better than having no paper but still way more work than just using it.
i do read "the morning paper" (https://blog.acolyer.org/) and he is really good in reevaluating papers. I'm very curious how much time he spends reading and analysing them.
> It is easy to miss that someone tackled the same problem as you (or had the same idea as you) last year
... and hard to resist the temptation to go jump out the window when you find out that the thing that you've been trying to figure out for a year has already been figured out...
There are independent benefits to struggling with something for a while by yourself before adopting someone else’s solution. For one thing, you will understand the problem and their solution a whole lot better.
Pretty much. There are for example studies that show that people who attempt a problem before finding a solution learn a lot more than people who just look at the solution.
The word research is pretty vague in comparison to the context the author interprets its meaning. He over-reaches on his explanation of what a researcher is, and leaves too much of a gap between researcher and tinkerer.
Maybe the distinction he is trying to make is between academic work and non-academic work (hence the emphasis on scholarship). Non-academics can do research but what makes it academic is its placement in the broader context of civilized endeavors.
> Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up.
I think its important to note that that he does reference NEAT (and the paper it comes from) and the emulator he used. This is building on what others have already done. While the 2009 paper from the post author would definitely be worth note, I feel "atrocious" is a bit harsh.
I tend to disagree with those distinctions and I'm inclined to think the difference is simply that, to a "researcher", doing things is a job; and to a "tinkerer", doing things is a joy.
## Scholarship
To begin with, there are many "tinkerers" out there who practise good scholarship and at acknowledge prior art, as well as provide references. Anyone who is courteous and has the awareness that the world doesn't revolve around her would honestly document these things even if she is a "tinkerer".
And just as there are "tinkerers" who practise good scholarship, there are "researchers" who practise bad scholarship as has been pointed out by @jacobolus.
## Testing
I'm not sure what makes the author thinks that rigorous testing is only a "researcher" quality. The article just insulted all the "tinkerers" out there by saying:
> "Here's another big thing. A tinkerer makes a thing and puts it out there. A researcher also tests the thing in some way, and writes up what happens."
A "tinkerer" doesn't just make something and puts it out there. A serious "tinkerer" would not want to put something that doesn't work out there—it's embarrassing and it's just, wrong. To make something (reasonably complex) that works usually involves a lot of testing, even if the goal isn't to publish a scientific paper it would still involve a lot of testing.
## Goals
This bit really lost me. It sounds like a bunch of self-contradictory, "researcher"-glorifying statements one after another.
> Usually, goals in research are not just goals, but ambitious goals. The reason we don't know what the results of a research project will be is that the project is ambitious; no-one (as far we know) has attempted what we do before so our best guesses at what will happen are just that: guesses.
A "tinkerer" can have ambitious goals. A tinkerer may do something that nobody has attempted before and, naturally, in that case she would just have to make guesses.
If, hypothetically, I set a goal to investigate the possibility of using CSS to make high-quality 3D games that can run at 60 FPS in web browsers, does that make me a "tinkerer" or a "researcher"? What if, in the process of doing so, I practise good scholarship, document everything carefully in a well-organised format, but just simply have no interest of publishing it in a scientific journal—does that make me a "tinkerer" or a "researcher"?
On this:
> However, if you read a scientific paper those are usually not the stated reasons for embarking on the research work presented in the paper. Usually, the work is said to be motivated by some scientific problem (e.g. optimizing real-value vectors in high-dimensional spaces, identifying faces in a crowd, generating fun game levels for Super Mario Bros). And that is often the truth, or at least part of the truth, from a certain angle.
The stated reasons in scientific paper is usually after written after the research has been carried out and they are designed to tell a reader why the works is important. In a space where everyone is struggling to keep up with publishing so that they can keep their job, who in the right mind would begin a paper with "our research group embarked on [insert amazing thing] because it seemed fun to do so and has never been done before"? Actually, I would love to read papers like that, because most of the time you know it's the usual standard crap when academics start a paper by stating a problem—it probably means that they haven't solved the problem and are not even close.
Also, I would like to draw your attention to the last, contradictory sentence. It seems that the author isn't quite sure either.
## Persistence
If a someone spends 10 years modifying and perfecting something that has never been attempted before, but does not care for its scientific value, does that make her a "tinkerer" or a "researcher"?
Bottom line, anyone who appreciates, and wants to do, good work will naturally do all of those things above—"tinkerer" or not. To throw away sentences like:
> Probably the most importance difference between tinkering and research is scholarship.
> A tinkerer makes a thing and puts it out there. A researcher also tests the thing in some way, and writes up what happens.
> While tinkering can be (and often is) done for the hell of it, research is meant to have some kind of goal.
> Tinkerers are content to release something and then forget about it. Researchers carry out sustained efforts over a long time, where individual experiments and papers are part of the puzzle.
The term research became extremely diluted now.
It attempts to put very different things under one roof,
and so is not very useful.
Near the end of that blog post you find:
>>Reading the book, I felt that most of my research is not science, barely engineering and absolutely not mathematics. But I still think I do valuable and interesting research, so I set out to explain what I am doing.
Of course, both the blog author and the book author are right in a way. But probably the useful thing to do here would be to come up with new names for these different kind of activities, rather than destroy the language by trying to just place one more thing into "research in general" category.
>Last year, some guy made an experiment with evolving neural networks for Super Mario Bros and made a YouTube video out of it. The video certainly reached more people on the internet than my work did; it makes no mention of any previous work. Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up.
Then publish your damn results in a damn open journal, dammit.
Most researchers in my field make all their publications freely available online after they have been submitted.
In machine learning and computer vision, the default is to put your work on ArXiv immediately upon completion. There is a lot of openly available research depending on the field. I find most fields in computer science are good for this.
In the case of the Mario example, I very much doubt that the author was not able to find other work because of closed journals.
Research can seem non-transparent to non-researchers because when problems are new, they are often poorly understood. Academic papers discuss novel problems and contextualize them based on other cutting edge work. Reading and understanding these papers requires a lot of context and takes time. Research is a skill that takes years to learn.
After some time has passed and we collectively gain a better understanding of a problem, academic papers may seem abstruse and overly complicated, but when these works were first published, this was the best way to understand them. For somebody looking for a recipe solution to a problem, an academic paper is likely not an the ideal place to look, which is why we write books, blog posts, etc... as we come to better understand a problem and its solutions.
> Then publish your damn results in a damn open journal, dammit.
Discoverability is hard. I mean, forget academic papers, take a look at just the web, and how many years it took Google to create a search engine that made discoverability easy, and that exploited (originally) the graph structure of the web. Papers do cite other papers, but how do you check a paper's relevance? Moreover, papers often describe something very specific, a specific approach, and searching for that approach is a lot harder than looking for a page that is "Haskell tutorial".
An open journal isn't the answer here (it is the answer for other things though imo) and I'm not sure why you're getting so angry.
This is one of those things that makes me deeply skeptical of the most zealous folks within the open science / open access movements. The emphasis seems to be much more on just throwing your data / paper up on the web sans context and much less on properly adding metadata and documentation for how to perform an experiment or how data can be reused. I've probably said this before elsewhere, but Josh Carp's "The Secret Lives of fMRI Experiments" really underscores this point. There are two equally catastrophic issues in the social (and to an extent) natural sciences and solving one (openness) is only part of the puzzle when the other involves blatant sloppiness in bookkeeping, pure and simple. If science is to overcome its current setbacks and lots of confidence, it needs to emphasize rigor alongside transparency.
All the metadata stuff is good, but I'm not in academia. I can't make use of it if it's not available. That's why I'm worried about the information being available first and foremost.
You also can't make use of data if it doesn't have appropriate metadata indicating how it can be used. Metadata is more than just "good"- it's essential for actually being able to run an analysis. Take fMRI for example- someone could just throw a brain image up on the internet. In fact, one particularly uninformed individual (who possibly broke Virginia state law) did so here: https://github.com/dcunited001/mri-scans
However, the raw data is not useful without knowing what scan sequence was used, slice-timing order, etc. Some of this is encoded in the data itself (the example above uploaded DICOM, which has somewhat rich metadata, although by no means is it comprehensive enough), but other data that is necessary is not (Phillips in particular is somewhat infamous for doing non-standard things with their scanners that aren't recorded in the DICOM headers, and in general it's a Herculean task getting MRI vendors to agree on standards). You could try performing an analysis with contextless data, but you're likely just going to make really spotty conclusions.
> I'm not in academia
Why should that matter? The circumstances of data collection are also important for business intelligence data or financial modeling data. I'm certain that the lack of this also contributes to poorly formed conclusions that cause real hurt to businesses out there.
Nevermind- I see where you're coming from now wrt not being in academia and not having access to data at all. My point still stands though- I think that you're grouping metadata and data together when you refer to "data".
Also the video discusses previous work starting at 4:40, so I'm confused what OP is complaining about unless the MarI/O guy re-uploaded with this attribution.
This is hardly a discussion of previous work. The author of the video states the paper he implemented.
In any semi-decent peer reviewed venue, you would cite a wide variety of papers that solve a similar or related problem or introduce a concept related to the method in the paper.
The related work section of a paper is one of the most important parts since it puts the research into context. By citing other work, the authors explain what has already been done, and what contribution their work makes.
A related work section should also illustrate the downsides, limitations, and differences of other cited research. Limitations of other works are often poorly understood since very few people have had the time to evaluate them beyond the initial experiments done before first publication.
Research is not simply about presenting new techniques but also understanding the trade-offs that arise when choosing different solutions to a problem.
Many times the works cited occlude (as opposed to illuminate) relevant research and the context of the paper. For example, if you read a paper and find details lacking, will reading the works cited help you understand the paper or just waste your time? Many times you won't have an institute license for the journal, so what now? Request interlibrary access, piracy, call a colleague at another institute, assume the arxiv preprint is equal to the published version cited if available, give up? If you cite too broadly, the value of the citations just go down for the reader, both for an expert who is probably already aware of the other research and for the novice who doesn't have years to read the entire paper trail.
This of course comes up during the peer review process as well. The referee informs you of a paper that is tangentially relevant, but you couldn't find that during your literature search because it is paywalled. How relevant was it actually to the work you performed or to people reading your paper if it is not readily possible for you (and possibly your readers) to access the paper?
Citations in academia are often times more the currency of the trade than actual scholarship. There is a reason review articles exist, not every paper needs to be a review article in its own right.
That is a different issue, though. The "Related Work" section is there to tell the reader about related approaches to similar problems, previous research on the same topic, etc. It's where you distinguish your work from what was previously done, and justify why this is a meaningful distinction. It's "what others have done and why we don't just use that".
Reproducibility is important but a completely different topic and I don't know why you're bringing it up here.
He mentions NEAT, which is good, but not any of the work on playing Mario, much less any work on evolving neural nets to play Mario. There's quite a few papers on the topic, not just mine.
Note that I don't say that SethBling is wrong. Seen as tinkering, this is perfectly fine, and he did a great job with the video. No hard feelings. I just use it as an example of how it is not research, because of the lack of scholarship.
Obvs he's complaining that the video didn't cite his work. :) One of the differences between researchers and tinkerers unmentioned in the article: vanity.
Like basically all researchers in CS these days, I keep all my papers freely downloadable from my webpage. Easily findable via Google Scholar, or just a regular Google search, for anyone who cares to look.
Yet, before today, I had never heard of you or your work. I download and read from a broad selection of papers and fields.
Mayhaps, you need to do a bit more "youtubing" to get your stuff out there and more accessible. A couple of my current favourites on youtube is 3Blue1Brown and Mathologer. They are both entertaining as well as being informative and have set me on investigatory paths for other research in various fields.
Why should they be familiar to you? There are hundreds of thousands of academics around the world working in thousands of distinct fields. Promotion on YouTube can be useful, but not nearly so much as you're thinking.
That's not always an option. Your career very much depends on consistently publishing in "A" venues (conference/journal). So it depends on your subcommunity's opinion on open access. For example, programming language people as a community are super keen on it, so most top venues are becoming open access. If that's not the case for your field, it's basically academic suicide unless you already have an amazing reputation.
You see, I read that clause in particular as "The difference between research and tinkering is whether you belong to an organisation that's willing to spend 30k+ a year on journal access".
And fuck that. If that's the attitude that academia is going to take, clearly we need to kill it off, tinker for a while, and start fresh.
Maybe we can refresh some standards for academic writing while we're at it.
I've heard some prominent academics speak to the same end, or farther, in the rare access I've had. (Yet with displaying no less passion for their chosen subjects. Who would have guessed!)
Freeman Dyson is actually quite vocal about his opposition to the current structure of academia, so I'm not surprised that you'd be able to find other academics who were equally self-critical.
I probably shouldn't be so surprised— but the number of esteemed academics I've had the chance to encounter I can count on one hand. Though many seem to share that same opinion.
The number of pretentious title-holders (even at the lowest minimum levels) I've met seems unending. I could get into that one... but I won't.
Instead you've just reminded me of yet more subjects to stack on my reading list.
This is all true. However, the large publishers (IEEE, ACM, Springer) these days all allow you to self-archive on your own webpage and typically also on ArXiv. So you can (and should) make your papers freely available. I have done that since the start of my career. It's just a matter of looking.
tl;dr Seth Bling directly references NEAT and his links[Show More [1]] link to 2 Togelius papers[13][14] among many others !
To wit, in OP, Togelius mentions Seth Bling's MarI/O[1] as lacking scholarship because he didn't mention Togelius:
"
seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up.
"
Rather disdaignfully (sadly/ ironically/ poigniantly) Togelius refers to Seth not by name but only as " some guy ".
Yet Seth Bling[2] in his short video, mentions and explains NEAT and directly links Ken Stanley's and Risto Miikkulainen's foundational NEAT paper[10] at U.texas which Togelius builds on [hint: click SHOW MORE under Seth's video]
And much more, i.e.: ---- the innovative toolchain Seth uses: Bizhawk to script SNES emulators in Lua [12]!
---- Seth links the wikipedia page on Neuroevolution[11] which directly references 2 of Togelius's own papers[13][14]
Seth's [Show More] Links on the MarI/O video[1] are very very comprehensive and (IMHO) indeed a great example of scholarship !
Togelius is a top flight scholar, and publishes in the field of AI game players and level generators[15] so I hope he will welcome some random tinkerer pointing out his error.
[full disclosure: I am a Togelius fan]
To give Togelius his due he does at least link to Seth's MarI/O video.
Perhaps Togelius failed to click / read the Show More links ? The charitable benefit of doubt principle is accidental oversight.
Seth Bling is primarily a Twitch Streamer[2] and Speed Runner[3] who often popularises complex programming[4], math and academic ideas[5] to a primarily young audience using Mario and Minecraft[6] and takes requests via his subreddit[7]
Seth's very popular Youtube video MarI/O which uses NEAT (Neuro Evolution of Augmenting Topologies] to learn and play Super Mario World, has led to a huge number of streamers on twitch playing other MarI/O levels[16], using RNNs, LSTMs, and other SMW levels and SNES games, like RNN multiplayer x4 Mario Karts[17].
The OP article is great, academic rigor and bibliographic scholarship might sometimes differentiate the full time university scholar from the amateur researcher, enthusiast or tinkerer.
Seth and his many followers may not be 'scholars' as in academic research but it is an amateur community of mostly youngsters doing their own research by having fun and sharing ideas, mostly undocumented outside their own transient twitch streams.
Compare this to the well respected amateur astronomy community and perhaps a responsible academic approach is to try to reach out and tap in rather than adding to the distance.
By his own OP standards of Togelius is lacking in that hasn't discovered/ mentioned the work done by the Twitch/ Bizhawk community - it is not entirely undocumented [16].
Togelius your AI Mario competition[8] is legendary in academic AI circles but not necessarily widely known by youngsters playing Mario ( and to be fair AFAIK you don't publish in their forums like twitch) although the code Infinite Mario[9] you base your work on is by Marcus Persson[10].
Though Persson is better known as Notch [10] ( author of Minecraft ) who is more famous than perhaps even Alan Turing , Yann le Cun, Jurgen Schmidhuber, or Geoff Hinton to the under 16 crowd.
Perhaps Togelius could mashup some of his work and do a crossover episode with Seth or relaunch the Infinite Mario competition[8] / benchmarks to this new younger crowd evolving Neural Topologies to play Mario games - or even help document some of their fun.
As public servants funded by tax dollars it is perhaps the responsibility of the academic to reach out rather than blaming fun loving amateurs popularising and doing research / 'tinkering' for not putting in the due diligence of scrupulously mentioning every source.
[IMHO] I agree amateurs don't always do full and proper scholarship and links are food for hungry minds - but they aren't publicly funded researchers but amateurs working for love of the subject in their spare time, for free, or for streaming donations and youtube hits, using transitory, live media and whose audience must be constantly engaged and may have shorter attention spans than the average academic.
[13] Risi, Sebastian; Togelius, Julian (2017). "Neuroevolution in Games: State of the Art and Open Challenges" (PDF). IEEE Transactions on Computational Intelligence and AI in Games.https://arxiv.org/pdf/1410.7326v3.pdf
[14] Togelius, Julian; Schaul, Tom; Schmidhuber, Jurgen; Gomez, Faustino (2008), "Countering poisonous inputs with memetic neuroevolution" (PDF), Parallel Problem Solving from Nature https://www.academia.edu/download/30945872/poison.pdf
Thanks for the commentary togelius [1, 2] and deepnet [above].
Anyone interested in adding a soupçon of scholarship to Seth's project?
Words do mean different things to different people, in different contexts..
There is a recent body of literature that explores the modern "maker" [3] movement. However, "maker" as a term may not have been a good fit for the OP's argument [1], which contrasted (academic) researchers with so-called "tinkerers".
An alternative term for "tinkerer" might be "bricoleur", a loanword from French. (Roughly, it still means one who tinkers: https://en.wikipedia.org/wiki/Bricolage but has other meanings depending on the academic lens.)
Originally known for work on _Perceptrons_ with Marvin Minsky, AI researcher Papert later adapted theories from education towards the vision of "learning-by-making" and the (young) bricoleur [4]. This can approach can be seen in the evolution from 1960s graphical [turtle] https://en.wikipedia.org/wiki/Logo_%28programming_language%2... to Lego https://en.wikipedia.org/wiki/Mindstorms_(book) to
modern day efforts to encourage coding-for-kids [5,6,7].
When it comes to early education, Turkel argues for epistemological pluralism [8] and cites anthropologist Levi-Strauss in comparing analytic science with a "science of the concrete".
We can appreciate both Seth Bling's concreteness [9] and Togelius's original papers for academics. Almost a decade ago, Togelius introduced Super Mario Brothers as a benchmark for reinforcement learning [10] and, with Karakovskiy, for AI more generally [11].
I have come to my personal conclusion that there is a difference between 'science' and 'the academic world' in much the same way as there is a difference between 'religion' and 'the church'. One can be very religious without going to the church and doing all the rites that are required by the church. In much the same way, a 'tinkerer' (or 'heathen') can be scientific without being academic. I believe that wiki summarizes it quite well for me: "Science is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe". There is no statement about giving proper credit (vanity) or not being allowed to reinvent the wheel without first making damn sure that you are indeed reinventing.
I want to solve real world problems and the way I want to do that is in a structural and repeatable approach. Science helps me doing this. Of course I'm not living in a vacuum, so I will look up as much as I can about the subject at hand as is reasonable within the available amount of time. Interestingly enough, I almost never end up with a research paper, but almost always with blog posts, books, tutorials et cetera. Products don't sell themselves and neither do research papers apparently. I think there is something seriously wrong if that is not perceived as a crisis in the academic world.