As someone who writes a lot of server code, I've always been curious about the architecture for MMOs--how do you handle issues of scale (or is it fine to assume everything is running on the same server)? If you do concern yourself with scale, presumably you have to "shard" your universe somehow? Do you shard by in-game geographical regions? If so, how do you handle players moving from one region (shard) to another? What is the best practice for handling game state? A SQL database? A redis server? Or maybe you just keep things in memory and persist asynchronously? Can anyone recommend blog posts or similar on these subjects?
Fun fact: the word "shard" in this context was coined by the dev team of the first major commercial MMO, Ultima Online, as an in-universe explanation for why the game world was split into multiple servers.
If you're looking for an exemplary server architecture for MMOs, look no further than UO. That game came out in 1997 and supported worlds with thousands of players and millions of persistent objects. Players could build their own houses within the game world, decorate (and later design) them to an incredible degree [1], and have other players visit (welcome or not). Players could run their own shops within their homes and sell their own hand-crafted, signed goods (weapons, armour, clothing, furniture) in addition to any found treasure they wanted to offer.
What made UO so impressive, technically, was how they accomplished all of this on such modest hardware as was available in 1995 [2]. This second link is to the GDC postmortem of the game. If you're interested, I recommend you check it out. It may not answer all your technical questions but it's extremely interesting nonetheless. The story of how they had to shard the game is part of the talk.
I'm pretty sure they did not use any off-the-shelf databases as the performance would have been terrible at the time. Everything would have been custom. The custom networking protocol is extremely reserved (not chatty), since it was designed for dial-up modems. Even when you run the game on modern hardware with broadband, the game uses on the order of a few megabytes per day of traffic.
I have heard Richard Garriott claim credit for coining "shard," and I'm very skeptical. The first database paper I could find that uses "shard" (as an acronym: System for Highly Available Replicated DAta) is from 1986: https://dl.acm.org/doi/book/10.5555/889956
Those authors may have been first, but I seriously doubt they were a significant influence on the term's popularity. If there's one thing I've learned from reading a lot of CS papers, it's that pretty much any word you pick out of a hat has been used as an acronym by somebody.
The paper you link to doesn't use "shard" as a noun, and in fact it doesn't describe a system that has "shards" in the sense that UO and just about everyone else since then has understood that term, i.e. independent partitions of a dataset:
> The reader is referred to [SBK] for a detailed description of the architecture of the SHARD system. Briefly, the main ideas are as follows. The network consists of a collection of nodes, each of which has a copy of the complete database.
(emphasis mine)
Additionally, according to Google Scholar, that paper was only ever cited 21 times. I checked the 16 of those citations for which full-text is available, and not a single one of them used the word "shard" as anything other than a passing reference to the name of the system itself.
For most Indie MMOs you can use a single server. You also store game state in memory and checkpoint it to disk like a DB, instead of having a DB as a bottleneck. This can scale to many hundreds of thousands of concurrent users if you do it right and it's easier than building a distributed system.
Scaling it beyond that (you're very lucky!) then involves sharding, and you can build a master/slave system that splits shards and moved them around as needed.
Source: working on my own MMO which is a realtime location based tower defense game.
All those questions are answered on github in what probably is the smallest (as a feature, not a flaw) MMO system codebase in existence. Short answer is I made my own HTTP server from scratch with distributed database over HTTP and everything runs on top of that. Sharding on separate machines is not interesting because the whole point is to play together, no matter how; you always want to be connected somehow. Larger teams separate the services and it becomes really complex. I want to be able to develop and maintain a massive MMO on my own so I had to improvise.
That said, I made a subreddit for the topic: https://www.reddit.com/r/mmodev/ it's kinda empty now but hopefully people will contribute eventually.
Persistence often uses much slower methods, one nice feature of my database is that it is distributed async. so I send all database queries to all regions which means I get backup and players can play anywhere without transferring data first.
I also use HTTP for real-time movement, you can do that if you strip all non-mandatory headers and use "Transfer-Encoding: Chunked" for the "Pull" socket. I use two sockets per client: "Push" (Client <-> Server) and "Pull" (Client <- Server).
HTTP uses TCP and that will inevitably lead you to the old TCP vs. UDP threads. My take on it is that TCP works fine for real-time action games today if you use event based protocols instead of tick based!
I know that "rules" say don't complain about downvotes, but man, I'm confused when people have downvoted me for complimenting someone's work and suggesting that you should support them. I stated it was $2 so that people who hadn't bothered checking out the Steam page would be aware of that, bu whatever.
@dang, I really don't think HN was intended to work like that. I watched a few minutes ago as someone went through my comments and over 30 seconds it went -2, -2, -2, which even makes me wonder if people have multiple accounts that they abuse for that.
Has anyone else experienced that?
Sorry for the meta, I just don't get that behavior. Kind of ugly.
edit: so yeah, same thing on this comment. Bam, -2. I dunno what spiteful game someone is playing, but it's silly.
People will downvote anything here. From facts to personal experience stories to complements, there are just people out there who downvote everything. And they'll get their friends to brigade too, just for the lulz. It's best not to worry about it, internalize it, or complain about it. They're just fake internet points after all.
With time, as technology peaks; more one-person engines will appear because you don't have to chase the next big thing all the time.
I'm building a MMO from scratch since 20 years back, the most recent MMO that uses my server is free-to-play this weekend:
https://store.steampowered.com/app/486310/Meadow/
The source for the backend is here:
https://github.com/tinspin/fuse
You can call it a one-man MMO backend.
I'm going to release a MVP for the C++ 3D MMO client in about a month hopefully, here is a tech demo for it:
http://talk.binarytask.com/task?id=579711216639635462