I will just offer some unsolicited advice about PhDs, as it came up a few weeks ago also. And the author touches on this. Of course this mostly applies to empirical sciences, but maybe some theoretical ones too.
An effective PhD advisor/thesis isn't wandering the woods to find something. It's a guided coaching exercise, with an outcome in mind. Test whether the PhD advisor you select knows this and has a history of taking this approach with former students.
An advisor (and the PhD research he/she guides you on) should be targeting a very conscious choice of incremental versus breakthrough research results. And for most students, you're not going to have a whole lot of mindblowingly field-changing results -- as much as you may think you're a rockstar. So it's just sensible or an insurance policy to make sure you've got solid steady work that you're progressing on.
Learn to enable your research to have visible incremental gains on a known path every day, rather than hoping for some breakthrough at the end. Amazing breakthroughs have high risk, and make it highly likely you'll have a crisis when it doesn't happen.
Concretely, even if you don't know what the answer is going to be at the end of your research, you must think about, or have an idea about, the format of what that amazing answer is going to be. Write the outline of your thesis and "ghost out" what the major charts will be. Write the intro sentences of each chapter -- what are they? (and I don't just mean the boring review of the field part, but your findings part)
You should know what major type of finding, plot, or table your research is going to output. What are the columns and rows of that table, or axes of that plot? How many data points are required? How many of them can already be guessed? Where is the surprise going to be? What is the conclusion going to be?
Draw out the answer you're aiming for, now. If you can't even articuate what the answer will look like, you may be in for a bad time, so work on fixing that. It will also push you and your advisor to be specific about what the output of your thesis will be -- and set you up for a much better PhD experience.
As a rising 5th year PhD in ML -- I could not agree with this advice more! I have very hands-off advisors. I spent the first 3 years "wandering the woods to find something".
Last year, I really had to sit down and think about how I can finish up my PhD on time. I pretty much did what you outline here. A lot of tools I used were organization tools I learned from my business school/product manager friends.
I honestly think the PhD system needs to be overhauled. even at a "top-tier" program like mine, it is amazing to me that at no point do we receive any training regarding practical components of research. 100 years ago, the way you became a physician was to follow around a physician and one day you were ready to be a physician yourself. In the year 2020, this is how PhDs are trained. I do realize that a PhD is not a "professional" degree like MD or JD, however, given that most PhD grads will 1) go to industry 2) go into academia, we need to teach students about project management, planning etc.
> at no point do we receive any training regarding practical components of research
I think the fact is that most advisors get no training regarding the practical components of being an advisor. So it really shouldn't be a surprise that there's a huge amount of variation here.
One my big takeaways from my own PhD was that different professors have wildly different levels of ability in terms of people management. All of the professors I met were very smart, but some of them were clearly very poor at the people management aspect of things.
Having said that, I'm not sure what I would actually suggest to a student going into a PhD program today. My observations were formed after watching a few projects crash and burn, and that's not necessarily the path I'd recommend for finding out if your advisor is a good one.
I had an advisor who just didn’t have a skill in or desire to advise. I didn’t discover this until in my junior year he went on sabbatical and the professor in transitioned to was really amazing. I still communicate with him 20 years later and hear about intern candidates, etc.
It’s one of those areas you need to know about — it never occurred to me to ask about or really look into what an advisor relationship can or should be!
> we need to teach students about project management, planning etc.
This is a problem with education in general. Those things should be taught in middle/high school as essential tools for modern life. Same as personal finances, effective strategies for team/group work, overcoming social anxiety, managing stress and building/maintaining relationships (personal and professional).
Project Management. I once was working with two CS professors in a UK university. One was full of words (let's call him A). The other was one of the brilllian minds in UI science that helped design the Nokia phones' keypads (let's call her B). Most people disliked B because she had a start, middle, end. She required timelines, milestones, deliverables. Because of that everyone working with her was getting better grades, was more productive, because she was helping people manage themselves better. Prof A had the lazy ones the "deep thinkers, but not the doers.
One thing that (many) academics lack is project management skills. This is why many (that I know) sit on a desk, with a mountain of papers, having a thousand things unfinished. Try doing that in an actual business and see how it plays out..
Being a good teacher doesn't make you a good project manager.
Project management is definitely very important. As a group/team manager where do you put the "deep thinkers, not doers" though? What is their place in business or society?
I personally identify with being a thinker instead of a doer and for most of my professional life struggled trying to find a position from where to contribute without feeling inadequate for not doing/executing.
I don't feel you "push them" to a specific slot. Thinking is good but something needs to come out. A book, a paper, a.. something. A number of people thinking for a common subject need to produce something, not think for 3 months and then start thinking of something else.
The PM skills is to get them to produce efficiently and effectively. Not perpetuate the "sitting and thinking".
Very true! And you can even be a good project manager and be nice about it. For example, setting clear expectations, working together toward deadlines, etc.
I'm not sure I agree. Everyone's situation in a PhD is different enough that the only general concepts you could teach would be high level and useless. Personal finance classes in high school are filled with junk like "make sure to spend less than you earn" and "here's how you balance a checkbook" and "never forget to pay your bills". For group work, the workshops I've taken for it talk about "listen before speaking", "repeat what the other person said so you are sure to understand it". Sure they're kind of useful, but mostly not.
You are saying that you've had bad experiences with courses trying to teach those things. Those things might not be easy to teach. But that doesn't mean they are useless or that they shouldn't be taught. To me, it means the methods of teaching those things, in the workshops you took, should be improved.
I think it's likely that there's no way to teach those courses that is useful to a broad population. It's just such a subjective thing, like imagine teaching a course on "how to be happy". Even someone trying really hard may have limited impact.
Couldn’t agree more. Instead of learning those things in school. I got lucky and had privilege and learned some of them outside of school, and still need to learn others.
But instead, they made sure I knew the name of the boats Columbus sailed in, in 1492. The Niña the Pinta, and the Santa Maria. I was forced to submit to whatever story they told me about the relationship between Columbus and the natives, no matter how untrue it was. But maybe that was the real lesson....
One of the reasons I recommend volunteer work to people who graduated or are about to graduate with nothing like a plan for getting a job is that this is how I learned a lot of organizational skills (and got my first references).
It also makes that, “what have you been doing for the last five months?” question a lot less uncomfortable to answer.
What do we cut out of the curriculum to find space for these things? We are barely able to teach people how to read, write and do arithmetic properly. I agree that these skills are essential, but it is so easy to just say that we should teach it in school.
I preferred wandering the woods. I transferred to a university that was more about wandering than coaching.
I'm skeptical of the calls to overhaul. Undergrads and even masters degrees went through this and now we have group projects and less individual learning. Too much focus on team work and collaboration. If anything we need to be teaching more wandering through the woods, more being on the out over the in of the group.
I would say there should be sort of a short PhD, maybe akin to MPhils, and ideally a step above the masters but below the PhD. The MPhils fulfil this ideal of more professionalized programs while PhD programs can remain more to their traditional styles.
Or bring back the Habilitation above PhD for serious academics. PhD is already sought after too much for professional reasons. Another 10 year commitment distills the lovers of wisdom from the lovers of cash.
Many years ago, I would have disagreed with you about the group projects. Today, I'm more in favour of them. The reason for my change of heart is that the moment you step outside acadaemia, be it industry in the same field, a research organisation, or any company, you'll be working on "group projects" as the default way of working. They are group efforts, and learning to work well in a group context is a valuable skill.
One big problem acadaemia has is the individual nature of research, even within research groups. A vast amount of time and effort is wasted on trivia due to working alone, which would be quickly resolved in a group. And while there is value in individual intellectual effort and reward, it can be vastly greater returns in a group context.
None of this is to say that lone investigation of out there ideas is bad. I think that's essential. But not for 100% of your time.
In many research settings, the scale of projects requires being a small part of a larger group. And while groups can squash good ideas, they can also kill bad ideas, and there needs to be a balance.
Not all PhD students are Einsteins, and it's as important to prevent people diving down a rabbit hole which leads nowhere as it is to give them the freedom to explore or else they might find they wasted several years rediscovering something already known or discovering nothing. I saw this happen to one of my fellows as well as myself. A good supervisor should be guiding appropriately. I found to my horror a PubMed search revealed a paper dating back to 1992 which had done several months of my research nearly two decades prior. When there are so many millions of papers, it's all to easy to miss stuff in the noise.
I had to leave the company I was at. Group projects were too confining. Work for myself now. The killing of bad ideas it just creates groupthink and encourages echo chambers. Some need it, some don't need it. I think the more we baby guide people into things without them discovering how to do things themselves the more they require the assistance of groups. The more they work in groups the less capable they are doing things on their own.
I have seen a trend in UK PhDs in the last 10 years or so to include much more of the "practical skills" type work. There's certainly been a big push around the "Researcher Development Framework" [0].
I would say that there's no one-size fits all for a PhD, but that I think something PhD students need taught how to do early on is identify their own development needs, and either address them, or seek the support to do that.
It's important that we don't turn PhDs into a taught course though, since ultimately the goal is to learn to direct your own research project from fruition to results, as an independent researcher. There's also a growing number of "structured taught first-year" PhDs (that then last 4 years rather than 3, with the first year akin to a "professional masters" with qualification at the end of it)
EngD degrees are probably a bit more like the JD or MD (if you're doing engineering, that is), but I don't have much experience of that approach. It does seem to be more along the lines of what you highlight a need for though (the project planning, management, etc.)
> I have seen a trend in UK PhDs in the last 10 years or so to include much more of the "practical skills" type work.
Gah I think this is a terrible idea.
In practice it means courses that aren't really relevant to anyone that you have to take when you really want to be getting on with your research.
Everyone's putting in minimum effort and getting nothing out of it. What a waste of time.
They're also taught by people slightly outside your field since they're taught across the department, so they teach you things that are seriously wrong for your field (I was told to publish in journals not conferences, for example, which is incorrect for my field).
They were always a frustrating waste of time. I just wanted to get on with my work.
It's only popular because it's aping the American PhD, and in my opinion the British PhD is better in practice (shorter, more focused, more grown-up, more independent, more professional) and should not be watered down.
I felt your way actually about it at the time. I like the 3 year timescale and greater level of self led work. I pretty much avoided doing any "taught" courses for the reasons you outlined, but I have found some of my students did benefit from them. I think it comes down to individuals and what they need. It wasn't right for you or me, but I think there are some students who perhaps come in needing a little bit of a push on some of the supporting skills, and they found some of the courses helpful.
I definitely don't like the trend towards the 4 year PhD with taught courses in year 1 though - I had enough time in 3 years to mess around on side projects and other things I didn't need, but which were fun and interesting, even if irrelevant. When you add the inevitable consulting and startup advice on the side, it seems to me 3 years should really be the upper bound, rather than extending the process any further.
I'm not from the UK. The idea of someone becoming an actual PhD in 3 years is rather quaint to me. Perhaps it's just a different approach, or perhaps the PhD system I'm used to produces a level of results that UK PhDs typically only achieve after their first PostDoc.
Or, perhaps there is an initial period of a year or so where the student is not yet doing a PhD, but is trying to produce results nevertheless. Saw that in Surrey, but can't remember how long students had after defending their PhD proposal and being promoted to PhD student.
In the UK you just get going on your research. You don’t do initial courses. You effectively defend your proposal when you apply.
Ultimately the goal of a PhD is to learn to be a researcher and to produce a good new research result. If you can do that in three years why wait around another two or more for the sake of it?
I had a colleague who did a PhD in two years in Austria. In that time he got two top-tier papers published. If you’re getting multiple papers into top-tier venues then surely you’ve past the test? You obviously can do research and you obviously are producing good results as judged by a wide group of peers.
Why does the US drag it out so much?
I’ll tell you why - US PhD students also only spend about three years on their PhD. They spend the rest of their time doing masters-level taught classes, teaching (!) and working on their advisors’ projects instead of their own!
Would you want to list those organization tools--if they are publicly available?
I find that a lot of important nitty-gritty knowledge is dismissed as besides the point, something akin to why MIT designed the "Missing Semester" course [1]. I would have a very hard time progressing with research if I didn't find Zotero [2], for example.
"it is amazing to me that at no point do we receive any training regarding practical components of research" - this varies A LOT based upon advisor. I'd be wary of drawing conclusions from a small data set. I've seen huge variances in what students learn from advisors, within the same department, within the same discipline, etc. It's too easy to paint "the PhD process" with broad brush strokes...
Part of my PhD programme was a "transferrable skills" set of courses which included planning and project management, Gantt charts and all that stuff to enable you to plan several years of work.
> An effective PhD advisor/thesis isn't wandering the woods to find something. It's a guided coaching exercise, with an outcome in mind.
There is a tradeoff here. One of Eric's selling points for his lab is obviously that students get a lot of intellectual freedom to develop their own ideas. This is the "wandering the woods" approach, which was my advisor's as well. The advantage is the freedom, as well as the responsibility and intellectual independence that comes from exercising it. The disadvantage is that it can lead to a lot of wasted time and dead ends.
The "guided coaching exercise", when taken to the opposite extreme, results in an advisor that hands a project with clearly defined goals and outcomes to the student. That kind of approach usually, in my experience, leads to more and higher-impact papers. It does not develop the student very well, but it does develop their CV, which is quite important, and once that student becomes a postdoc, it is do-or-die time, and every CV item helps.
But if we just view this advice as "how should someone doing a project conceptualize their project", this is good advice. Although you would be astonished how many PIs don't follow it. I've had PIs want to write papers before seeing a single figure. I've had countless collaborators send me data to analyze without any clear idea what sort of results/analysis they want, without a hypothesis even.
The job of a PhD advisor is to guide the student to become an independent researcher. That is: someone able to conceive of research-worthy ideas, separate the good from the bad, execute the research and write it all up in a paper.
I have seen very hands-off approaches and very structured approaches. Both have their merits and pitfalls: the first may leave a student stranded in the proverbial forest, while the second may turn into nothing more than a 4 year long series of related tasks to be executed, set at day one, with no input from the student.
On the other hand, I've also seen both approaches work well.
Basically, you want a daily supervisor that cares, and with whom you have a click.
> Test whether the PhD advisor you select knows this and has a history of taking this approach with former students.
How do you "test" for this? Asking potential advisors "do you know this?" is blunt and rude. Asking potential advisors in a mannered way prompts anodyne answers. Current students in the potential advisor's lab are unlikely to offer an intimate assessment of their personal satisfaction and research progress to a stranger, assuming they can even be honest with themselves.
My take is that having a good graduate advisor is largely about having a good undergraduate advisor who can offer an insider take and advocate on your behalf.
To add a data point, my advisor told me to do the same. In fact, I actually spoke to one of his then-RAs who was leaving to go to the other side of the world (by sheer coincidence).
That discussion confirmed I was making the right decision, and I was able to gauge as well from the "warnings" I received that I was going down the right route. So I would definitely echo the above, and advise you speak to at least one of their previous students. If none are still around, that might be a red flag (or they might all be getting snapped up right away into jobs, but they will make the time to discuss with you, if they felt their advisor helped them!)
I would personally find it surprising if someone who wanted me to advise them didn't seek views from those I had advised previously.
I know of two examples of professors who wrote a detailed syllabus like yours, albeit in different styles. One was my brother's advisor, and the other is my brother. ;-) Perhaps not coincidentally, these professors were also known for having a straightforward approach, and for their students getting through the process reasonably unscathed. That's not to say it was easy for students, or that conflicts didn't ever occur. But there was little or no confusion about the expectations.
You are definitely not every PhD advisor. This is a public service and should be required reading just about every institution that issues advanced degrees.
I would say that as a student, you would be asking a potential advisor what mix of projects would be an example course of the research. What is "their approach" to the sequence of your grad projects over several years? Examples? What combination of risk/certainty does it have? I think you can tell pretty quickly whether the advisor has this concept even in mind.
Former students should have this idea in their mind too, after finishing the program. If they don't, they haven't learned one of the important things about a PhD...
And if you feel that it's too rude to ask, or the professor can't handle such a question (or you're hesitant to ask) -- either you probably want to find someone who is comfortable with such questions, or better find out why you are uncomfortable. Better to be awkward up front and get the answers you want, than be polite and suffer for 5 years to discover the wrong answer.
I mean, that's a whole nother topic -- this is a huge investment of years of your life. You should ask difficult questions before you make the leap. And the professor should owe you such answers before committing to taking you on.
In fact, it's not that different from an awkward pre-nup contract conversation. Only this time you don't have to be swayed by emotion.
Admittedly approaching this with a non-US perspective, but it seems strange to me that PhD students approach their advisors to see what the project/approach is.
Is it normal (in the US?) for a PhD to be as "guided" or prescribed by the advisor as you suggest above? Perhaps my experience differed, but I went for an advisor who takes an old-fashioned "independent research" focus. That's not to say they were unavailable or disinterested; rather they were there to offer input and guidance when needed, but not to steer or even direct. It was up to me to plan what I wanted to do, when to do it how to go about it, what to write up and publish, which conferences to bother with etc. Perhaps this is unusual, but I felt it was a key part of the experience to deliver your own major research programme.
I'd definitely agree that if you can't ask the questions you outlined above, or if they can't answer it, that is a big red flag. A PhD is about learning to ask questions fundamentally, and then set about answering them, so may as well get started by asking your advisor. I'd also try to get an understanding of the culture of the group and practices, since some research groups have more of a "lab" culture where everyone works as part of a bigger project, while I was involved in the opposite - everyone had their own "thing", but would work together when it was helpful.
Well, my story was 15 years ago and in an area where students were mostly expected to bring their own research questions to the program. I had trouble getting the time of day from potential advisors, to say nothing of interviewing them about potential projects and their advising methodology!
This is brilliant in a way that relates to commercial research in natural science too, generalizing some of the terms and focusing more toward business than academic life:
1. What difference will this work make if you succeed?
a. Whose lives will be made better by this research?
-the client
b. How will this improve upon what’s currently being done?
-the client will be more satisfied
c. Why is this one of the most important questions in the field?
-because a valuable client asked
that was easy
d. Will it create a big policy change at some level (company, government)?
-if so it will be for the positive with consensus, otherwise no costly changes
e. Will it inspire a new class of systems?
-would be good if it was worth building beyond clients' immediate needs
2. Who will care about it when you’re done?
-two clients would be better than one
a. Will government agencies care?
-only in a beneficial way, when the FBI comes to your office you want to be their consultant not their suspect
b. Will platforms and industry care?
-if you do projects which make them money, more fondness can be expected than if you cost them money
c. Will non-commercial researchers care enough to recognize or benefit from it?
d. Will non-commercial researchers care enough to teach it to their staff/students?
e. Will anyone care about it 10, 20, 50 years in the future?
3. How will this change what other people (defined broadly) are doing?
a. Will other researchers change what they’re working on after seeing your work?
b. Will practitioners do something different?
c. Will users adopt what you’ve made, found, created?
d. Will authorities use what you’ve found to draft new requrements/specifications?
-unless no negative impact can be imagined, such a sensitive project might best be shelved until a time of more positive outlook
Professor Gilbert has an _internal deadline_ 7 days more strict than the real publication cutoff, actually allowing for reduced stress and making higher Quality output possible in ways others may never achieve:
>If the paper isn’t ready, we won’t submit it to that conference or journal deadline; we’ll submit it somewhere later.
Naturally deadlines for being AT the conferences are not missed. I bet they can build anticipation about future presentations most accurately when this happens.
My impression is that things are drastically different if you’re going for a top professorship vs something else, like a job at google. If you’re going for a professorship then basically nothing matters until you strike on an idea that hits it out of the park in some way. And it’s a tricky combination of technical and political fitness. Depending on your work, some schools will like you more than others.
If you’re just looking for a job in some research lab then a basic “get it done” process on an incremental result works fine.
> Concretely, even if you don't know what the answer is going to be at the end of your research, you must think about, or have an idea about, the format of what that amazing answer is going to be. Write the outline of your thesis and "ghost out" what the major charts will be. Write the intro sentences of each chapter -- what are they? (and I don't just mean the boring review of the field part, but your findings part)
> You should know what major type of finding, plot, or table your research is going to output. What are the columns and rows of that table, or axes of that plot? How many data points are required? How many of them can already be guessed? Where is the surprise going to be? What is the conclusion going to be?
You are absolutely right and it's depressing.
The only practical, predictable and steady way to a PhD is to come up with the storyline in advance, do the experiments, bend the interpretation and analysis to support the story you already wanted in advance, then pretend the finding is surprising and novel (when actually you knew you could safely make it work in advance).
There is a tension here, on the one hand PhDs have to pretend play that they discovered major novel stuff and innovated a lot (which is only possible through immense risk), on the other hand in reality they must finish in a given number of years and must be very risk averse.
Only a major cultural shift towards handing out PhD titles for meticulous, systematic, risky but unfruitful research may help. Forcing people to find flashy new stuff will lead to tons of overinflated findings that are more about storytelling than actual surprising novel things that bring a field forwards. It's all a huge pretend play.
----
Ideally in an alternate universe of different incentive structures that support real science, a person would work in research, use existing approaches in different scenarios, push and prod them in different use cases then perhaps once every year you'd stumble upon some interesting effect you don't understand. Then you'd dive in and prod this specific effect with well designed experiments. Once it's clearer what is going on, you come up with the best way to convey this newly earned knowledge. It is at this point (once you actually know what you want to alert the research community to) that you decide how to best present the knowledge, what tables, sections and paragraphs will succinctly and ergonomically convey the information.
If you knew all the story and layout beforehand and you know you will be able to just plug in the numbers and make the paper stand, then you are not doing real science, you are playing the academic game, a cargo cult of science, a game stealing and consuming the prestige of science.
Yes, as I said this only works in the alternate universe. Here in this universe we must churn out papers and must overinflate our findings. Of course, we need to eat, we need jobs, we need titles etc.
It has driven so many PhD students I know to bitterness and cynicism. Several have given up and went to industry. Others, after the disillusionment, have cynically pushed through with whatever minimal things they knew would be required (plus the right story and presentation) then bailed to consulting. You also need to bullshit in consulting but are at least compensated for it well enough.
In the rarest cases do people come out of it as a positive passionate person. Even the successful ones become a hedging, scheming, lukewarm, slimy academic personality. Everything encourages this. The way you get professor positions, the way you get grant money etc. It optimizes and chisels people into becoming the antithesis of curious, open scientific inquiry.
>In the rarest cases do people come out of it as a positive passionate person. Even the successful ones become a hedging, scheming, lukewarm, slimy academic personality. Everything encourages this. The way you get professor positions, the way you get grant money etc. It optimizes and chisels people into becoming the antithesis of curious, open scientific inquiry.
Got a point there, all kinds of academic pressure other than the push for scientific progress.
There should be more than one way to outperform what you could do in such an environment, or in the industrial counterparts following that pattern.
>Several have given up and went to industry.
I think worked for these bozos before, not fun but milestones are milestones.
>Ideally in an alternate universe of different incentive structures that support real science, a person would work in research, use existing approaches in different scenarios, push and prod them in different use cases then perhaps once every year you'd stumble upon some interesting effect you don't understand. Then you'd dive in and prod this specific effect with well designed experiments. Once it's clearer what is going on, you come up with the best way to convey this newly earned knowledge. It is at this point (once you actually know what you want to alert the research community to) that you decide how to best present the knowledge, what tables, sections and paragraphs will succinctly and ergonomically convey the information.
Well, I think one of the shifts that happened is that research changed from "I'm charting my own course in an undiscovered field and accept all liability" to "a university factory promises me a degree and experience".
But here there is an incomplete jump -- in some programs it looks like a well-charted degree program, but has all the liability of research that may not work out. That's why I think (as pointed out above) some practical engineering fields do much better than purely theoretical fields in this regard.
It has also become much more expensive to give it a try and be wrong.
People who don't understand these factors and go in with the wrong expectations are highly likely to fall through the cracks, and not recognize when their program is going sideways or their advisor/topic relationship is not working out.
The illusion that we are supposed to uphold is that brilliance and serendipity can be planned for, can be scheduled and made into an academic program. Actually research and scientific discovery should never be the job of someone. It should be a side product of general activity in a field, something that is already useful even without the discovery (if you're poor). Or just lesure hobby activity that is fine if nothing comes out of it (if you're rich).
It's also part of the whole inflation and treadmill thing going on in education and titles. Generations ago, going to high school was something significant. For the next generation, high school was the default and college became a special thing. Now the majority (in the US) goes to college. Within the family it feels like satisfying progress. Parents look on their kids and feel proud that they are making it further than they themselves did. But now a Master's degree can be a bit too generic. Having a PhD can be nice in industry nowadays (though of course not required).
So what used to be a special thing for a limited circle of select few with the background and fallback that allowed failure, is now becoming a factory process. PhDs are produced by the thousands and thousands, and we pretend they all pushed science forward. There is a flood of papers accompanying this. Tiny steps forward or mixing and matching existing things, but selling a big story around it, flag planting, trying to claim a huge area when only a tiny aspect of it is actually demonstrated, drawing out huge conclusions etc.
There are many reasons for this. The general societal inflation is just one thing. The other is the overall quantification, standardization and uniformization in our zeitgeist. There must be a standard process for everything. The politicians and bureaucrats demand to have a standard process for innovation. You must know in advance exactly "how many pieces of innovation" will you produce per year, what will be the impact of it etc. Then when you tell them about your plans, all they hear is the number of buzzwords mentioned from the latest fads and tune out otherwise. Ah and they look at your publication list of course. If you churned out many papers in the past, you will probably do that as well in this round of funding, and those papers can all be attached to the funding agency reports, so that will make the agency look good towards the ministry or whatever.
> If you knew all the story and layout beforehand and you know you will be able to just plug in the numbers and make the paper stand, then you are not doing real science, you are playing the academic game, a cargo cult of science, a game stealing and consuming the prestige of science.
I actually agree with your immediate statement here, but that is not at all what I understood the OP to be saying. I read their "You should know what major type of finding..." argument as being one of starting with a concrete and well-formed research question, and carrying out carefully designed experiments, and thought that it was excellent advice.
In my little corner of computer science, I very frequently see people (at all stages in their scientific careers) start working on some new bit of research by a) getting a bunch of data, which they then b) feed into some nifty model du jour, and then c) spend a ton of time overcoming all manner of technical trials and tribulations, then finally d) get a number out the other side. They then e) find themselves totally stuck when it comes to actually interpreting their result, because before they ran their "experiment" they hadn't actually bothered to formulate a concrete hypothesis, and so it's not clear what they are supposed to _do_ with their shiny new number, or where to go next.
That's what people often don't get about science, whether it's wet or dry. The mechanical process of actually performing the experiment itself is usually the easy part, relatively speaking. The hard part is thinking carefully about the thing you're trying to study, formulating a theory, coming up with testable hypotheses, and designing experiments to perform those tests.
Part of that last phase involves planning ahead very carefully to what you're going to measure, what your control and intervention criteria will be, what specific statistical analysis you'll perform on the resulting data, and what your various interpretations will be. The more concrete and explicit you can make this, the better: "We're going to measure 'X' under conditions 'A' and 'B', because we think that 'X' will be a valid/useful measure of $PHENOMENON_WE_CARE_ABOUT, for reasons ____, ____, and ____. If X_A ends up being bigger than X_B, our interpretation will be ______, and if X_B is bigger than X_A, we will instead conclude ______'; if they are the same, that will suggest ______." [1]
Obviously you don't yet _know_ which of those conclusions you'll be drawing (if you did, it wouldn't be an experiment), but it is absolutely essential that you've gamed out the various possibilities to at least this level of detail _before_ you do the experiment. This is doubly true for exploratory analyses where you don't really have an intuition about what the outcome will be, as it helps keep the analysis from turning into an endless fishing expedition ("Well, what if I normalize this variable _this_ way? OK, what about _that_ way? ...").
In my experience, one of the best techniques for doing this is, yes, to actually write out blank versions of the tables that you think you'll need to tell the story of your experiment ahead of time, and to make dummy sketches of the various figures you'll need to help interpret the data. Not only will this help you clarify your thinking about what you are hoping to learn from doing the experiment, it has the added benefit of making sure that whatever code you write actually logs/outputs all of the needed data elements! More than once I've had to re-do an experiment because there was an important piece of data that I hadn't realized I would need until it was time to do the analysis. With just a bit more prior preparation, that poor performance would have been prevented.
To return to the OP's argument, they weren't saying that you should pre-specify your conclusions (which would be a terrible idea, for the reasons that you clearly spell out in your post). They were saying that you should have a plan about what specific experiments you're going to run and _how_ you're going to describe the motivation and results of those experiments.
And, if I may editorialize for a moment here, having a more structured approach to doing and writing about research can go a long way to helping to reduce the angst that comes with doing a PhD. I do very much think that many CS PhD programs are dropping the ball in terms of teaching experimental design and evaluation- but that is a rant for another time, as my TED talk today is already running long enough. :-D
--------------------------------
1: Very, very, very often, the process of formulating things this way takes several iterations, because usually once one is forced to write it out this explicitly, all sorts of little questions pop up- "Wait, is that actually what it will mean if X_A > X_B? What if means ____ instead? Hmmm... maybe I should be measuring X', instead? Oh, I'll need different data, in that case, because..."
An effective PhD advisor/thesis isn't wandering the woods to find something. It's a guided coaching exercise, with an outcome in mind. Test whether the PhD advisor you select knows this and has a history of taking this approach with former students.
An advisor (and the PhD research he/she guides you on) should be targeting a very conscious choice of incremental versus breakthrough research results. And for most students, you're not going to have a whole lot of mindblowingly field-changing results -- as much as you may think you're a rockstar. So it's just sensible or an insurance policy to make sure you've got solid steady work that you're progressing on.
Learn to enable your research to have visible incremental gains on a known path every day, rather than hoping for some breakthrough at the end. Amazing breakthroughs have high risk, and make it highly likely you'll have a crisis when it doesn't happen.
Concretely, even if you don't know what the answer is going to be at the end of your research, you must think about, or have an idea about, the format of what that amazing answer is going to be. Write the outline of your thesis and "ghost out" what the major charts will be. Write the intro sentences of each chapter -- what are they? (and I don't just mean the boring review of the field part, but your findings part)
You should know what major type of finding, plot, or table your research is going to output. What are the columns and rows of that table, or axes of that plot? How many data points are required? How many of them can already be guessed? Where is the surprise going to be? What is the conclusion going to be?
Draw out the answer you're aiming for, now. If you can't even articuate what the answer will look like, you may be in for a bad time, so work on fixing that. It will also push you and your advisor to be specific about what the output of your thesis will be -- and set you up for a much better PhD experience.