This is basically a re-implementation of the recent work from OpenAI[1], where you first collect high-quality observation-action-observation transition tuples, train a model to predict the actions between observations, then use that trained model to label a much larger observation-only observation dataset (e.g. youtube videos of people playing a game).
Finally, you just do dead-simple behavior cloning (predict "expert" action from current observation) on the large labeled dataset. It's still surprising to me how well this works! Behavior cloning has some theoretical issues since it doesn't address the sequential nature of decision problems at all, but apparently with enough scale it can still do fairly well - super cool.
What they did with Dota 2 was really surprising, the AI did a technique where if it was ahead it would go behind the enemy tower to get 100% of the gold while the enemy would struggle with last hits under the tower and take creep damage. I'd never seen anyone do this before and pro's started copying this after. The way I remember them explaining it was that the AI has something it's trying to maximize for, and it plays millions of games trying slightly different variations which can lead this kind of creativity, pretty wild.
Fascinating! You can see it is trying to mimic many gameplay tactics used by CS:GO players, but it's just barely missing the mark. It's like a monkey trying to mimic human behaviour.
Examples:
- Follows the wall contour using the crosshairs when moving around the corners. The only thing wrong here is that the AI is aiming at the wall instead of at the open space next to the wall where the enemy would be.
- After going through a door, it checks all corner to clear the room. But it's very slow in doing this, and would get shot instantly in a real game.
- Adjusts the crosshair constantly to not look at the floor when walking. The problem is that the crosshair is never at headshot level!
I am not bashing on this AI btw, just pointing out that it's actually getting close to real human behaviour! Fascinating!
Something I've noticed is going well is the fact that the AI doesn't spray n' pray but controls the spray pattern and drags the crosshair down. Shoots in bursts!
> The only thing wrong here is that the AI is aiming at the wall instead of at the open space next to the wall where the enemy would be.
This is actually how well-practiced humans play, aiming instead where the enemies are likely to be (behind the wall) instead of trying to perfectly aim at the corner all the time.
It depends on how you’re planning to peek. Generally you’d want to smoothly track edges to clear out everything, including off-angles. Only if you’re quite certain of an enemy’s position you’ll “aim at the wall”, then peek out. Your cross hair will then be spot on immediately. This method always runs into the risk of overpeeking and getting killed easily. Smoothly tracking edges has the risk of getting swung out wide on, instead of getting peeked at tightly, such that sometimes you want to place your cross hair a bit away from edges, but only if you’re the one holding an angle statically.
Excellent. Believably plays like a pro with regard to spray patterns but maybe slightly slower. This is doom for any cheat detect that relies on outliers. Doesn't look aimbotty in the slightestt.
Well, it's not public knowledge but it seems that VAC does not care about how it looks. Also while it doesn't look aimbotty, according to modern standards it also sucks at the game. In contrast, many pro players look very aimbotty, but are legit. But ultimately yeah, anticheat will be totally lost.
Interesting. What's stopping the AI from more or less instantly detecting human shapes and then dishing out headshots instantly as well? Technically, that should be possible. I suppose its learning material is strictly actual humans, so it'd never be able to learn that, even if technically feasible.
There's already bots that are capable of this (often used for cheating in games) - but this research is trying something more difficult and interesting - which is teaching an AI how to play a game, with nothing but a ton of learning material.
Things like moving around the map in a rational way, learning how to counterplay enemies, tactics when entering a room, all based on solely visual data, are extremely difficult. These skills are much more intuitive to humans, but much harder to learn than "there is a human head at {523,1021}"
Aimbots have been around in FPS games for a very long time, including purely out-of-band approaches (observe video output, send input) as they cannot be detected by anti-cheat software.
They started with a system that lets players review games and try to understand if a player is a cheat or not. Games are anonymized, and you get to rewatch the replay to try to make a determination
They can then use that data to train their NN to help detect which players / games should be fed into this manual review process
I assume now the product is probably good enough that it's ranking reviewers too to determine how good reviewers are at detecting if there's cheats or not.
It's pure genius
Here's a GDC talk you may find interesting:
> In this 2018 GDC session, Valve's John McDonald discusses how Valve has utilized Deep Learning to combat cheating in Counter-Strike: Global Offensive.
kinda a missed opportunity they scaled the whole screen down to 280x150. They should have left the cross hair area larger--so like 10x10 around the crosshair unscaled, and instead of doing weird warping geometry, just pass it in as a second 2d image. I'm guessing the focus of the research is on the behavior cloning, which is really awesome!
> Whilst RL research often aims to maximise reward, we emphasise that this is not the exclusive ob-
jective of this paper – perfect aim can be achieved through simple geometry and accessing backend
information about enemy locations (hacks and built-in bots exploit this). Rather, we aim to produce
an agent that plays in a humanlike fashion, that is fun and challenging to play with and against.
I’m very ignorant when it comes to AI but I’d like to have a machine watch me select some sub strings out of different sequences of text and have it find what is the pattern behind all the choices I make. Is there something like that I can train?
That sounds a bit like the SQuAD 2.0 challenge, but simpler.
In SQuAD 2.0 a model is given a question, and a passage of information. It has to select the start and end points of where the answer is contained in the passage.
Your problem is simpler in that there is no question - you just have to have a model that can identify the start and end positions that _you_ will select.
Sounds like excel flash fill might be similar to what you’re interested in. It’s a well documented feature described in papers and talks, it comes from a pretty cool group in MS research.
My data is a 600 char string of letters and following some very loose rules you slice 7-10 substrings of the string. The string is an RNA sequence and the substrings vary depending on the genus/species/etc. I know loosely how far down the string each might be, but it’s not perfect and it varies. I definitely know in what order they are, so that helps. Right now I have a script that works 90% of the time, but I’d love to make it learn from the data it reads so it can guess the other 10% or be ready for new discoveries. So I’m not sure how flash fill would work in this scenario.
it has already been a couple days, but here is my $0.02 (for context, i have played cs:go for 3k+ hours):
this game has much more going on than games like mario, and even dota 2. for one, audio matters! also, there are more inputs to train, such as walk, jump and crouch.
this is a great start, but a lot more needs to be done before it can be called a complete solution.
Still, heard of people getting banned. Or if you accidentally leave something on and swap accounts. I just don't want any software or modification on my PC that could be seen as cheating.
They could take a 20 year old that has never played fps's and perform similar tests. Coordination, twitch muscle reflexes, peripheral vision and other variables will never exist with AI. GLHF as they say.
Hopefully not too off-topic, but AI playing chess, doing this or that "better than humans" is not a 1st world problem.
I never followed the chess scene and was intrigued with the Han Niemann angle, but noticed the chess engines being overblown with a lot of it (play ping pong against a wall type of stuff).
The 1st thing I thought was a chess format called "turn the tables" where chess players perform the 1st 3-5 moves, then "turn the board" and compete with the oposing players setup (did not research and this may already exist).
A Deep Blue versus Garry Kasparov rematch using "turn the tables" gameplay would be cool.
I did laugh at Magnus Carlsen winning in 1 move when the opponent resigned after a crappy knight move by Magnus at the recent tournament. TtT chess takes a lot of the memorization strategies out of play for the natural GM's.
With much of this AI stuff, nobody is chasing AI because of its severe limitations, impediments and attribute deficiencies. Just play ping-pong against a wall.
Finally, you just do dead-simple behavior cloning (predict "expert" action from current observation) on the large labeled dataset. It's still surprising to me how well this works! Behavior cloning has some theoretical issues since it doesn't address the sequential nature of decision problems at all, but apparently with enough scale it can still do fairly well - super cool.
[1]: https://openai.com/blog/vpt/