Hey! Great project. We actually built stdlib [1] to capture an experience similar to Heroku's for building microservices, and we'll be presenting at AWS Re:invent on Tuesday [2]. Our workflows with our public API are accessible via our open source library, would love contributors such as yourself (and anybody else that finds it interesting!). We have a diverse team working on the project - Xooglers, support from early Heroku team members, and even some YC alumni lending a hand. :)
I'm wondering if it would be possible to create something like Lambda, but optimized for multiplayer servers with game loops? What if there was a platform that let you write pure functions to create player accounts, spawn game instances, and handle the transformation of player and game state from tick to tick? (You'd have to choose a particular architecture for synchronization with the client, of course. Also, before someone makes the mistake of assuming I'm as stupid as they are -- I'm not proposing that a function be transmitted, compiled, then evaluated at each tick over the network! Functions would be installed into virtualized servers.)
Clojure, Javascript, and Golang would be good languages for this. The environment possibly would let you inject and replace functions live, lending to a Light Table like experience, but while developing a multiplayer game.
Your basic description sounds like LambdaMOO (written in 1990, the name is a coincidence).
The LambdaMOO server has a main event loop which handles user events, then schedules tasks which execute functions ('verbs' in LambdaMOO parlance) to modify the game state. Verbs are not pure per se but can be thought of as a transaction which executes in an atomic, consistent, and isolated manner. Verbs are attached to objects in the game, and can be modified in-game on the fly by players with their programmer bit set. (Programmers can only modify verbs on objects they own.)
The LambdaMOO language is a lot like JavaScript with a Luaesque syntax. It has prototypical inheritance, which meshes well with its concept of objects as physical things. For example, to get a new instance of a duck one would create a Generic Duck named "My Duck" ('Generic' being the conventional prefix for an object specifically intended to be used as a prototype.)
There was a time in the late 1990s when it regularly supported 300 active users at once (with a certan amount of lag); on modern hardware it could probably handle significantly more than that.
If you want a modern scalable system then LambdaMOO itself is probably not for you; the server is single-threaded (in fact the whole thing is remarkably similar to NodeJS with async/await support) and state is saved in-memory and checkpointed periodically (hence the Durability missing from the above description of verbs).
However, the basic design (prototypal inheritance, in-database verbs, ACI[D] tasks, etc) is extremely well proven, and would lend itself well to being copied into a modern distributed system like AWS Lambda. As long as you had an ACID datastore to back it up, you could scale the task execution as far as you wanted, and with a bit of clever work you could probably come up with a partitioning scheme to make the database scale as well.
In fact, I've partially implemented this already; what's stopped me has primarily been that I'm more interested in the technical aspects and have no idea what I would actually use it for once I built it. If you have a use case for this sort of thing I'd love to hear about it.
There was a time in the late 1990s when it regularly supported 300 active users at once (with a certan amount of lag)
I had a pseudo-roguelike server first implemented in Clojure, then in Go that updated 12 frames a second. It could support 250 simultaneous users. It even had Conway's life as an area attack. I had it posted to Show HN.
However, the basic design (prototypal inheritance, in-database verbs, ACI[D] tasks, etc) is extremely well proven, and would lend itself well to being copied into a modern distributed system like AWS Lambda.
All that's needed is pure functions, soft realtime, and good tooling and APIs. ACID takes some doing, in terms of implementation and will also make scaling complicated, as you mention above. Inheritance? Nice but not necessary. Just let developers inject functions into a game loop, then tell them their load and when their simulation tick rate starts to drop.
If you have a use case for this sort of thing I'd love to hear about it.
Yes. Something like that could be packaged into MMOGAAS -- Massively Multiplayer Online As A Service.
Different ones have different APIs tuned for different types of gameplay and or Unity3D integration (by mimicking Unity's original networking programming API).
I've implemented them for an MMO I freelanced for a while back and it was pretty painless.
So Photon is operating a farm of servers that are running game loops, which can have pure functions injected into them? (Also, the per message pricing!)
No, Photon is doing "Massively Multiplayer Online As A Service" (did you even read their page?).
And actually they are running game loops, but you can't inject code into them unless you run their software on-premise, or use a separate system I recall them mentioning that offered server-side scripts that got injected in their loops.
And cost per message make perfect sense because of the networking model, which is fully message based. Paying per message is paying for bandwidth + cpu usage as a single combined rate.
No, Photon is doing "Massively Multiplayer Online As A Service" (did you even read their page?).
And actually they are running game loops, but you can't inject code into them unless you run their software on-premise
So you are seemingly contradicting yourself. 1st you said that "they are doing that." Now you are saying they are also doing "Massively Multiplayer Online As A Service" -- but in a quite different way than what I specified. So yes, I did read what you posted. Apparently you only carelessly read mine.
I think you need to read my post again then. I'm not contradicting myself in any way.
You acted like no one offers a PaaS for MMOs (and gave it a ridiculous name).
I pointed out just one of many PaaS options for MMOs (or ...MMOAAAS as you insist on calling it.).
In fact, when working for that MMO we did evaluate options the did let you use their game loop with your cloud functions being called on triggers, but you'd have to look that up yourself, that was a few years ago (even Photon has added new services since I did that research).
If you think you have an approach that can enter the space, by all means go ahead, but PaaSes for MMOs have existed for ages.
I didn't log in for a while and my account was deleted circa 1999, I was pissed, since I'd had it since about 93, although there was little I could do about it. What a great mud though. :)
I only discovered it around 2012, so I never got to see it in its heyday. I had great fun programming different things, though I haven't found the time in a couple of years. I still log in occasionally just to wander around and marvel at things--there's all sorts of weird stuff lying around, though I can never seem to find something when I go looking for it. I don't think they reap accounts anymore, since there's no longer a massive influx of players putting pressure on the database.
If I was building a MUD or MMO from scratch today I'd try with Elixir (the language built on Erlang). Functional programming with hot code replacement, implementing the actor model for distributed, fault-tolerant, soft-real-time massively concurrent applications, does that sound similar to what you're looking for?
Growing that into a FaaS product would be hard though; Erlang processes all share a trust domain, from the infosec point of view, so containerizing each function/process for invocation lambda-style on otherwise shared infrastructure may prove difficult.
Each of my server processes in Go looks a little like an Erlang actor. There's a switch statement where a queue of input messages is processed, then each instance is run in turn. Instances also have a similar structure and maintain a "space" using an R-Tree. When it comes to efficiency, game loops are still a great way to do concurrency. Game loops per-instance allow you to sidestep many pitfalls of multi processor concurrency, much as per-process GC does with Erlang.
You could, but the latency issues (network hops) would make the solution untenable. Function as a Service platforms are ideal for solutions where scalability is a huge concern, latency not so much. These functions are all running inside of containers that have been optimized to be "warmed up" extremely quickly, but that warm-up takes time, and the more robust the solution, the longer the warm-up.
You'd probably want architecture extremely specialized to a game loop, and probably your game loop specifically, if you wanted low-latency, high-availability FaaS for this purpose.
You could, but the latency issues (network hops) would make the solution untenable.
...ok... Did you read my comment?
These functions are all running inside of containers that have been optimized to be "warmed up" extremely quickly, but that warm-up takes time, and the more robust the solution, the longer the warm-up.
Yeah, of course that's not going to work.
I've basically built a specialized "container" system, but instead of using kernel virtualization, it's actually a specific Go "object" residing in an already running server process. These "containers" (really instances) start running in an environment where we already have a virtual server with a process with a server/game loop up and running. I have a cluster of processes that does this, and idempotently spawns an instance of star system in a multiplayer game consisting of 2^87 star systems.
www.emergencevector.com
Such a system can still be built on Docker style containers, it's just that those Docker style containers represent an entire sub-cluster of many server processes that can each host many "instance" containers which are where the functions that transform from tick n to tick n+1 reside.
You'd probably want architecture extremely specialized to a game loop
That is what I said in my comment.
and probably your game loop specifically
I'm saying that the game loop could be customized/built up by a developer by injecting functions into such a system. (Or, possibly, by injecting such functions into a dev version running locally, then exporting them to a server in the cloud.)
This is great. I was going to create something similar to manage lambda functions across environments. Awesome to see you're already using the newly released environment variables for lambdas.
I didn't plan to do the push-to-deploy thing, mostly because that would totally complicate things. The rationale was more that we would use the CI that would automatically build and deploy things to DEV env that would be then promoted around.
Has anyone characterised server less architectures? I'm interested in testing / exploring the limits with some predictive modelling work and implementing (mathematical) functions as FAAS? No doubt, it 'depends' (e.g. nature of data and its locality or otherwise) but interested to hear if anyone is looking at very fine-granularity server less systems?
Serverless is way much more advanced machinery than this tool.
My point was to create a tool, not a whole ecosystem around AWS Lambda.
When it comes to serverless - my main concern (as it was with Lambda without env variables), is with storing configuration data along the code in the repo. Additionally the reason for creating was mostly pipelines.
Very cool project, but it has me thinking what/if it would be possible to build out more of the infrastructure management that Heroku has to make it even easier (domain setup, addon store, etc.)
Looks cool! I like the release message stuff, I wanted to add similar to apex(1) at some point, not having a clue what the releases actually are in Lambda makes rollbacks a little tricky.
You acted like no one offers a PaaS for MMOs (and gave it a ridiculous name).
I was talking something specifically like Lambda, but where pure functions could be injected into an already running game loop. The name was intentionally ridiculous. Also, you suggested that my proposal would suffer from the latency of lambda requests going over the network, which shows you didn't read my original comment.
with your cloud functions being called on triggers
Yeah, I pretty much covered that this was not what I was proposing, when I wrote "(You'd have to choose a particular architecture for synchronization with the client, of course. Also, before someone makes the mistake of assuming I'm as stupid as they are -- I'm not proposing that a function be transmitted, compiled, then evaluated at each tick over the network! Functions would be installed into virtualized servers.)" As far as I knew (which might be out of date) Photon was developing something that would be suitable for a persistent world MMORPG, but not quite ready with it. What I'm proposing would be suitable for an MMORPG, not just MMO game sessions.
I frankly don't care what further squirming you feel like doing to convince yourself you have a novel idea
Excuse me, but where ever did I say that I had a novel idea? I merely pointed out that 1) I wasn't talking about exactly what you thought I was talking about (and after 4 iterations, you still don't get that) and 2) you went and attributed to me a bad idea I had explicitly disavowed in the 1st place.
Actually I was so annoyed by your insult that I had to look you up Peter.
You're clearly a real class act and a careful reader.
I guess you mean the API-Gates + Lambda. Not sure - I think thats something that should be handled by some separate tool (APIGatesgger? ;) - lets hope its not another dirty word)
that's right swagger -> api gateway triggering lambda functions.
I was using Chalice but seems like Zappas is another one a HNer pointed out. I couldn't get Chalice to work and sort of gave up on the serverless-api-gatway-lambda idea.
[1] https://github.com/poly/stdlib
[2] https://www.portal.reinvent.awsevents.com/connect/sessionDet...