This looks neat, but anyone interested should know there's also a whole mature field to solve similar problems. The field of control theory has some really simple and really robust methods for getting a system from state A to state B. If you aren't trying to learn how the system works simultaneous to your algorithm being deployed, it can be a really a good fit.
This book and control theory solve different problems. I'm just commenting because I sometimes see modern ML folks trying to apply super duper general techniques to problems that are easily solved specifically with simple undergrad-level techniques from the specific fields (including this material to control problems). Also, I just think control theory is really cool and perhaps under-discussed in "cool math/eng stuff" circles.
You are right that sometimes people use much more sophisticated algorithms than are really required to solve a problem. My experience is that the simplest approach is often the best one in the long term. There are actually deep connections between problems in control theory and the topics in this book (e.g., LQR control), though these different communities often use different notation. The focus of the book is more on higher-level decision making, but the execution of the decisions---such as motor commands to an actuator---are often best done through PID or some other method that can be found in a control theory textbook.
Control theory is indeed, quite cool, and under-discussed in most circles.
Also, unfortunately, classical control theory is primarily good for linear time-invariant dynamics in the frequency space of the Laplace transform. If you can't locally linearize your model and/or need to learn a model, classical control approaches are underdeveloped, and everyone has switched to optimal control and RL.
I think you might mean something different by optimal control than I would. I'd bucket that into control theory, but it seems you're drawing a distinction.
I would distinguish between classical control and optimal control. In the former, you don't have a (Hamilton-Jacobi-)Bellman equation to solve, while in the latter, the approximate dynamic programming is The Hard Part.
This is neat - specifically because what you point out is actually true. The primary author is a professor and the secondary author is a Phd graduate - both in the Stanford Aeronautics and Astronautic Department [1][2] - in the Dynamics and Control Group.
A brief skim through of the book shows many references to Control Theory texts.
That explains why there is not as much traditional Decision Theory in the book as one would expect from the title, such as Peter C. Fishburn's work, issues in MAUT and MCDM, seminal work by Bouyssou, Vincke, Pirlot, Keeney, or the older discussions about SEU vs. EU, evidential vs. causal decision theory, etc.
It's a rather technical book, focusing on issues like advanced sensor data fusion, control theory, and optimization. Not that there is anything wrong with that, the title is just slightly misleading.
Ben Recht [0] has worked on control theory and reinforcement learning.
See his blog post "What we've Learned to control" [1] and the survey paper "A Tour of Reinforcement Learning: The View from Continuous Control" [2] may be of interest.
I learned it from a course where my professor had his own notes printed by the school bookstore for us, and I've held onto those notes like they're gold. So I can't refer you anything by firsthand knowledge, but the internet seems to like Dorf & Bishop's Modern Control Systems, which seems like a decent coverage of the basics:
the next step after PID control is state-space based control.
it's not a single tool but more of a framework. it goes beyond frequency based representations and can also model non-linear control problems.
the kalman filter is an example of the use of this framework where it's combined with statistics.
the problem in control theory though is that once you want to go beyond linear control things get very difficult, most of the literature seems to be about finding clever ways of approximating your problem into something linear.
Check out model predictive control. And LQR if you haven’t learned that yet. And the bazillion state estimation techniques that start with Kalman filtering variants (eg UKF) and progressively account more and more for nonlinearities.
Kalman filters and variations are covered in Ch. 19. It can be viewed as a special case of belief updating in a partially observable Markov decision process.
I bumped into this accidentally having ever heard of it. It is one of those pleasures of browsing through a library. Lovely book, easy to understand with a broad range of topics and insights. Recommended!
The title reminded me of this anecdote (which I can't locate now, might have been somewhere in Kahneman's or Ariely's recent oeuvre):
This associate professor of Decision Theory received two offers of tenure track positions at reputable universities, and got some friends and colleagues together to discuss which one to take. One of them suggested he use the tools of his discipline, Decision Theory, to help him make a decision. To which the professor replies: "Now, come on guys, this is serious..."
Thanks for that source, a talk by frightfully brilliant Stanford mathematician and magician Persi Diaconis. It contains another cute anecdote:
> One of my most endearing memories of the great psychol- ogist of decision making under uncertainty, Amos Tversky, recalls his way of ordering in restaurants: “Barbara? What do I want?”
The first author (MK) also wrote "Algorithms for Optimization" (MIT Press), which is also implemented in (and is a nice guide to) the Julia language, which is itself a pretty sweet language.
Does anyone have any advice on how to make the most out of books like these? I'm trying to read more textbooks in subjects that I really phoned in during college this year. My current strategy is to make some note cards and take notes (which is kind of tedious), wondering if anyone has any advice as to their own workflows.
My suggestions, not for this book in particular but technical textbooks in general:
- Get very familiar with the table of contents first. Then note and filter out the sections you are less interested in or don't appear to provide the biggest bang for your buck.
- Annotate, annotate, annotate. Active reading is better than passive reading. I use a 2-in-1 chromebook and a passive pen to highlight in the adobe pdf reader.
- Write detailed notes in addition to annotating (i.e. in a separate notebook or markdown file). Cull/convert your notes into Anki/Memrise digital flashcards. Drill on your phone when you are lying in bed.
As a programmer I usually try to implement some of the algorithms in language of my choice and throw them at some toy problems (it's good there is no shortage of interesting toy problems in this field). It was really fun with Artificial Intelligence: Modern Approach book (ex: implementing a SAT solver from scratch and solving Minesweeper by pure logic is very satisfying).
1. Ten pages a day
2. Do all the exercises, and redo all the ones you get wrong.
3. Add anki cards for all the key terms, ideas, formulas, etc. The best textbooks include a summary the end of every chapter that you can double check your notes / anki against. Prefer cloze markup to question / answer cards -- more cards per "fact" but longer retention.
4. Double check your cards against a source like Wikipedia for completeness / accuracy, since sometimes there's an author bias to content with.
This takes about an hour a day for #1 and #2, and an hour a week for #3/4 plus a few minutes a day extra in the Anki review queue. For this book, you're looking at about 2 months start to finish.
Get in a group. Work is a good place to do this. Pick out some books, send a mail round saying that you are running a tech book group. Get folks together and read a chapter a week (this is hard in reality - be cool with everyone and sometimes let chapters slip over a few weeks). Every week someone leads the discussion. Every week someone volunteers to do the next weeks. You meet for an hour to discuss.
I enjoy typing, so I like to simply type out the entire textbook. It's a bit ridiculous to some, but it slows down the pace that I actually learn and engage with the content. A textbook usually takes me a couple months to type, and I'm never in a rush though finishing chapters is satisfying and an easy goal for an evening. I find it similar to as if I were to attend a lecture on the topic.
David Goggins makes the claim that due to his learning disabilities he acquired the diving instruction manual 18 months ahead of time and wrote it out by hand 10 times in order to learn the material.
As somsone who's currently trying to translate an instruction manual from one language to another, it is... somewhat fragmentary. Yes, you read the text, multiple times (and in multiple languages).
But, the focus is not on internalising the material, the focus is on producing something that makes sense in a different language.
I guess part of the problem is that the source material I am working from is from 1596, in blackletter, and pretty archaic "source language", so the actual process is "transcribe original", "translate transcription".
it's hard for me to learn something if i don't have something I need to do with the knowledge. it's not that I can't sit down and do it, but things won't stick in my brain.
so I would say, try to find something you want to do that requires the information in the textbook. or else just don't bother reading the textbook and go do something else you actually want to do.
I typically take a book like this and read it. Then I think about it. I also think about it as I read it. I ask my friends in person and online if they've read and talk with them about it. Sometimes I'll look for online forums which discuss the book. This is how I make the most out of books like these.
This looks very good from my skim! It includes state-of-the-art approaches like Actor-Critic with MCTS and loads of recent and interesting things like GAIL and SMILE. The areas where I have expertise superficially seem like good treatments. I'll have to see what other RLers think and perhaps use it to study up on belief states :)
Much more up-to-date than sutton and barto, but the authors are rather less well known.
Mykel Kochenderfer is my uncle! He does some really cool research up at Stanford. Lots of autonomous quadcopter drones doing automatic maneuvers and learning how to not crash into each other. It’s cool to get to see his work on HN!
Since you had an Ask HN regarding machine learning resources, and your uncle has written some machine learning papers, have you considered asking him to teach you or is he really busy?
I have talked to him about it, actually! He gave me some great course recommendations that I plan on looking into. Ask HN was mostly motivated by wanting to get multiple data points/opinions.
I dont know about you, but I have been trying to cover this topic for some time, but it is very difficult to assimilate. Like many, i am a self learned developer, I do have higher education in sciences and the math is tough to me but I can understand what they are talking about. Still those algorithms are challenging
this book describes the linear programming formulation for finding the optimal value function. cool to see, I think this formulation is underrated by the CS community compared to other communities also trying to solve MDPs.
however I wish it also described the dual of this linear program. this problem involves optimizing over state-action frequencies which is equivalent to optimizing over policies.
so value functions and policies are dual to each other. that's pretty neat! not sure why modern RL texts don't talk about it at all.
Thank you for posting this. I think the titel is a bit misleading. When reading "Decision making under uncertainty" I would expect some follow up on Decision Theory by Kahnemann, Keeney, Roy and Brans and methods considering uncertaity in e.g. AHP, Electre, PROMETHEE etc.
In some ways this touches on some of the same issues raised in Superforecasting by Philip Tetlock and Dan Gardner. Really interesting to think that we could methodically make better predictions and decisions and potentially greatly improve our society and quality of life.
Lets say I wanted to verify a student read a page of text from a novel. And I wanted to automatically generate a question from that text. Any algorithms that can currently do that?
If you wanted to do Cloze questions, you can use saveall.ai (a tool that I built) to generate them automatically from fragments of texts. We also plan to work on automatically generating questions from a larger chunk of text, but it is very difficult. Some people have had some success with GPT-3 but it's unreliable
It's a minor nitpick, but anyway, what's up with naming your pdf files like this 'dm.pdf'? What's wrong with 'Algorithms_for_Decision_Making.pdf' of even adding the 'Mykel_Kochenderfer,Tim_Wheeler,Kyle_Wray_-_' prefix?
I can't put a proper name in a "save" dialog until I open the book, and I need to save it first!
I sound so annoyed because I recently downloaded 'hist.pdf', 'fxtbook.pdf' and 'V090212S.pdf' Is it just to get a memorable file URL? If so, humanity invented simlinks long ago.
Wild guess is that the main document LaTeX file is called "dm.tex", so the resulting PDF would be called "dm.pdf".
Making the main file named something more verbose would impact you whenever you try to open it, but in an ideal world, you'd rename it once it gets to the point of it not needing many changes.
I very much doubt that the book ends up on a web server as a result of some LaTeX-CI rather than being uploaded by a person.
I'm trying to be considerate and provide a uniquely identifiable file name whenever I make some available to the public. In my opinion, books require the full name and revision, and, as I've already mentioned, you can have a link to it (or a copy) with a short name if you'd like to have an option of a short URL for verbal link transmission.
This is a very important dimension. See Sec. 1.5 of the book. There is also a side reference to a book that discusses this and other societal implications.
This book and control theory solve different problems. I'm just commenting because I sometimes see modern ML folks trying to apply super duper general techniques to problems that are easily solved specifically with simple undergrad-level techniques from the specific fields (including this material to control problems). Also, I just think control theory is really cool and perhaps under-discussed in "cool math/eng stuff" circles.