I use Game Theory in pretty much everything (academic)
Here are some thoughts on the literature.
There are several different strands and evolutions of Game Theory.
1. Game Theory (non-cooperative):
The basis was Neumann/Morgenstern Theory of Games. It has been suggested in this thread, however its focus is a bit obscure today. Still useful for repeated games, for example. Both authors are also important for Decision Theory, see below.
Afterwards came Nash, defining the what the basic solution concept would be up until maybe 1990. Simple Nash equilibria are used primarily where rational agents choose in mathematically nice spaces where uncertainty is not a major factor.
Following Nash, the Game Theory literature developed to produce equilibrium refinements.
These, usually subsets of Nash equilibria, were created because Nash often predicts very little - the space of equilibria is often so large that nothing can be learned, or uncertainty requires the incorporation of different information sets of agents.
The first developments came while incorporating uncertainty and multi-stage games (where people move in sequence).
Harsanyi was able to show that most configurations of uncertainty situations can be represented as a Bayesian Game (the issue was the recursion of "he knows, that I know, that he knows that I know..."). The problem became, that these often produced unintuitive and large sets of equilibria.
So we have refinements. Some target robustness, like Selten's Trembling Hand. Others target "natural behavior", empty threats and so on.
Almost all of those refinements are a subset of a Nash concept. The development of refinements was en vogue prior to the 90's, when it stopped for reasons I will detail below. Basic Nash has survived, however, and is still the go-to tool to understand multi-agent decision problems (at least initially).
1.a Cooperative Game Theory:
Largely in parallel, mathematicians like Shapely and later economists like Roth also tried to think about cooperative games. Here, we don't look directly at what individual people do in isolation, but rather what groups are stable and plausible and what they can achieve. If for example a smaller group can "break" a coalition, then such a large coalition can not be considered a plausible solution. Matching theory comes from here, for example, so you will find it in most problems of assignment (say, students). Much as non-cooperative Game Theory, it is applied widely.
2. Decision Theory:
Decision theory developed in parallel and is a wide field. It is, however, critically important to Game Theory because it sets the stage for information, constraints and decisions that agents take. Expected utility, by Neumann and Morgenstern, was and is the basic instrument to understand how agents incorporate their knowledge. This was based on objective probabilities, so in parallel the Bayesian stream also developed. With a monumental and beautiful proof, Savage then developed Bayesian Decision Theory (based on works by de Finetti and others). This is critical to many, many fields in maths, statistics and science in general, and was then the basis for Game Theory. Aumann is associated with latter refinements of decision theory.
Later on, the idea of uncertainty (Knightean uncertainy) became important. This is when you can not assign a probability to an outcome. Paradoxes by Elsberg and Allais have shown that this is actually an important decision problem in real life. Multiple approaches exist to generalize Decision Theory, such as Prospect Theory, MinMax Preferences, integration by capacities as opposed to measures. Schmeidler, Gilboa and Wakker are some names.
Game Theory exists in this space as well.
3. Evolutionary Game Theory:
The idea came from Biology and is important because it is a way to justify Game Theoretic outcomes without even requiring purposeful action by agents. It had a huge impact on many problems, especially dynamic ones and "top down" models, but did not surplant traditional Nash in general. Some scientists believe it should. Other's think it's just one more tool. There are those who believe the whole of social sciences should be based on it... Let's say it did not achieve that yet.
4. Economic applications:
Economics was historically the discipline to apply Game Theory most. Initial concepts like Nash justified many early models of Markets. Earlier concepts of non-perfect competition were formalized with Game Theory.
Things really started to take off when asymmetric information were introduced. Think Moral Hazard, Signaling Games, Contract Theory and so forth. What we know about economics, organizations, business, competition and many social phenomena today has largely been developed by applying Game Theory. There are too many great names to mention: Akerlof, Tirole, Spence, Hart, Homström, Myerson, Stiglitz.
5. Mechanism Design and Auction Theory
In the 70's and 80's, from the above applications, economists like Hurwitz, Myerson and Maskin developed mechanism design.
The idea is simple and genius: If agents play games, what if we can choose the game they play? Which game do we choose without them walking away, but with us getting the desireable outcome? What is, in other words, the optimal mechanism inducing the agents to play a game?
Initial examples and todays shining example of econ in action is Auction design. Which sort of auction mechanism is best to sell ads, be ebay or assign broadband licences?
Mechanism design leads to very complex problems, which is why until the early 90's many simplified assumptions were used. While mechanism design has been very useful, this also lead to two developments. In econ, papers started to get more and more complex to accomodate real life issues like non-monetary transfers, dynamics, complex type sets and so forth. Computer scientists trying to implement mechanisms quickly discovered that many were simply to complex, so they started Algorithmic MD.
6. Experimental and behavioral games:
So earlier, I said that a whole cataloque of equilibrium refinement basically died out. Why is that? Well, with behavioral econ we were introduced to more realistic approaches to decisionmaking. Then questions arose, such as "what if I can not count on rationality of my competitor". As it turns out, this may actually break the inference of Nash equilibria pretty handily.
At the same time, economists and psychologists put people in experiments to play games. In some situations, Nash worked well. In other situations, one could accomodate much by using more complex Decision Theory.
But in many instances, people would just not play Nash. In other words, they couldn't even figure out the most basic solution concept. Indeed one can do all sorts of experiments in a Game Theory 101 class showing that people often choose much too heuristically.
Equilibrium refinements make Nash more complex, it was clear they had to be abandoned. Currently, research joint in decision theory and game theory works on finding better ways to model behavior when Nash is not reached.
Books:
Osborne/Rubinstein has been mentioned. Contrary to what was said, this is an undergrad book and a solid intro.
There are two classic works. The major one is by Tirole and Fudenberg, the other is by Myerson. The former is more standard, the latter is better.
Now there is a new book by Maschler, Solan and Zamir with like 900 pages. It's really good, and I would definitly get it as a second book after an intro.
For Mechanism Design, the best book is by Tilman Boergers. It's also free to download.
Auction Theory specifically has a standard volume by Krishna.
Both of those are math heavy. This is true in general, but Game Theory concepts can often be explained by intuition. For Mechanism Design, I fear that a solid math background would be required, because the space of "choosing a game" is mathematically not so nice. However solid means you should have a good grounding in analysis and optimization, perhaps dynamic systems. Basically, a math heavy undergrad education will be fine.
There are several different strands and evolutions of Game Theory.
1. Game Theory (non-cooperative):
The basis was Neumann/Morgenstern Theory of Games. It has been suggested in this thread, however its focus is a bit obscure today. Still useful for repeated games, for example. Both authors are also important for Decision Theory, see below. Afterwards came Nash, defining the what the basic solution concept would be up until maybe 1990. Simple Nash equilibria are used primarily where rational agents choose in mathematically nice spaces where uncertainty is not a major factor.
Following Nash, the Game Theory literature developed to produce equilibrium refinements. These, usually subsets of Nash equilibria, were created because Nash often predicts very little - the space of equilibria is often so large that nothing can be learned, or uncertainty requires the incorporation of different information sets of agents. The first developments came while incorporating uncertainty and multi-stage games (where people move in sequence). Harsanyi was able to show that most configurations of uncertainty situations can be represented as a Bayesian Game (the issue was the recursion of "he knows, that I know, that he knows that I know..."). The problem became, that these often produced unintuitive and large sets of equilibria. So we have refinements. Some target robustness, like Selten's Trembling Hand. Others target "natural behavior", empty threats and so on. Almost all of those refinements are a subset of a Nash concept. The development of refinements was en vogue prior to the 90's, when it stopped for reasons I will detail below. Basic Nash has survived, however, and is still the go-to tool to understand multi-agent decision problems (at least initially).
1.a Cooperative Game Theory:
Largely in parallel, mathematicians like Shapely and later economists like Roth also tried to think about cooperative games. Here, we don't look directly at what individual people do in isolation, but rather what groups are stable and plausible and what they can achieve. If for example a smaller group can "break" a coalition, then such a large coalition can not be considered a plausible solution. Matching theory comes from here, for example, so you will find it in most problems of assignment (say, students). Much as non-cooperative Game Theory, it is applied widely.
2. Decision Theory:
Decision theory developed in parallel and is a wide field. It is, however, critically important to Game Theory because it sets the stage for information, constraints and decisions that agents take. Expected utility, by Neumann and Morgenstern, was and is the basic instrument to understand how agents incorporate their knowledge. This was based on objective probabilities, so in parallel the Bayesian stream also developed. With a monumental and beautiful proof, Savage then developed Bayesian Decision Theory (based on works by de Finetti and others). This is critical to many, many fields in maths, statistics and science in general, and was then the basis for Game Theory. Aumann is associated with latter refinements of decision theory. Later on, the idea of uncertainty (Knightean uncertainy) became important. This is when you can not assign a probability to an outcome. Paradoxes by Elsberg and Allais have shown that this is actually an important decision problem in real life. Multiple approaches exist to generalize Decision Theory, such as Prospect Theory, MinMax Preferences, integration by capacities as opposed to measures. Schmeidler, Gilboa and Wakker are some names. Game Theory exists in this space as well.
3. Evolutionary Game Theory:
The idea came from Biology and is important because it is a way to justify Game Theoretic outcomes without even requiring purposeful action by agents. It had a huge impact on many problems, especially dynamic ones and "top down" models, but did not surplant traditional Nash in general. Some scientists believe it should. Other's think it's just one more tool. There are those who believe the whole of social sciences should be based on it... Let's say it did not achieve that yet.
4. Economic applications:
Economics was historically the discipline to apply Game Theory most. Initial concepts like Nash justified many early models of Markets. Earlier concepts of non-perfect competition were formalized with Game Theory. Things really started to take off when asymmetric information were introduced. Think Moral Hazard, Signaling Games, Contract Theory and so forth. What we know about economics, organizations, business, competition and many social phenomena today has largely been developed by applying Game Theory. There are too many great names to mention: Akerlof, Tirole, Spence, Hart, Homström, Myerson, Stiglitz.
5. Mechanism Design and Auction Theory
In the 70's and 80's, from the above applications, economists like Hurwitz, Myerson and Maskin developed mechanism design. The idea is simple and genius: If agents play games, what if we can choose the game they play? Which game do we choose without them walking away, but with us getting the desireable outcome? What is, in other words, the optimal mechanism inducing the agents to play a game? Initial examples and todays shining example of econ in action is Auction design. Which sort of auction mechanism is best to sell ads, be ebay or assign broadband licences? Mechanism design leads to very complex problems, which is why until the early 90's many simplified assumptions were used. While mechanism design has been very useful, this also lead to two developments. In econ, papers started to get more and more complex to accomodate real life issues like non-monetary transfers, dynamics, complex type sets and so forth. Computer scientists trying to implement mechanisms quickly discovered that many were simply to complex, so they started Algorithmic MD.
6. Experimental and behavioral games:
So earlier, I said that a whole cataloque of equilibrium refinement basically died out. Why is that? Well, with behavioral econ we were introduced to more realistic approaches to decisionmaking. Then questions arose, such as "what if I can not count on rationality of my competitor". As it turns out, this may actually break the inference of Nash equilibria pretty handily.
At the same time, economists and psychologists put people in experiments to play games. In some situations, Nash worked well. In other situations, one could accomodate much by using more complex Decision Theory. But in many instances, people would just not play Nash. In other words, they couldn't even figure out the most basic solution concept. Indeed one can do all sorts of experiments in a Game Theory 101 class showing that people often choose much too heuristically. Equilibrium refinements make Nash more complex, it was clear they had to be abandoned. Currently, research joint in decision theory and game theory works on finding better ways to model behavior when Nash is not reached.
Books:
Osborne/Rubinstein has been mentioned. Contrary to what was said, this is an undergrad book and a solid intro. There are two classic works. The major one is by Tirole and Fudenberg, the other is by Myerson. The former is more standard, the latter is better. Now there is a new book by Maschler, Solan and Zamir with like 900 pages. It's really good, and I would definitly get it as a second book after an intro.
For Mechanism Design, the best book is by Tilman Boergers. It's also free to download. Auction Theory specifically has a standard volume by Krishna. Both of those are math heavy. This is true in general, but Game Theory concepts can often be explained by intuition. For Mechanism Design, I fear that a solid math background would be required, because the space of "choosing a game" is mathematically not so nice. However solid means you should have a good grounding in analysis and optimization, perhaps dynamic systems. Basically, a math heavy undergrad education will be fine.
hope this helps