Wow they didn't use any DB at the start of UO, that is amazing haha
"As Raph also notes, there were no databases originally involved in the storage of game state or player data for UO (disregarding analytics here), everything was kept in flat files. Backups worked by flagging a moment in time where no one was allowed to cross server-boundaries -- during that moment, each areaserver was commanded to fork(), essentially duplicating itself in memory (it's more complicated than this, thanks to Copy-on-Write, but let's simplify). After everyone had fork()ed, the "lock" preventing boundary-crossing was cleared. Then each areaserv began to dump out its huge chunk of memory-state into a file on an NFS server. Those files were then all tarred together and kept as a "backup" of the state of the server. These heavyweight backups happened at half-hour intervals, I believe."
I'm not so shocked... I'm a web developer, not a game developer, but databases were avoided as part of production stacks in the 90s. Most people tried to stick to flat files. A database layer added complexity and decreased performance. Even most content management systems had an option to write everything to a flat file to serve via a web server. Those were the days that Apache servers and a huge pile of flat files was the norm.
Once web content began to be personalized, that was no longer realistic. Databases became a required part of the stack, web servers ran applications instead of just serving file, we gained massive amounts of functionality, and haven't really looked back.
It is, but it adds the requirement for installing and configuring another application running in the background. Before SQLite, flat files really seemed like the best option for small projects.
I don't understand what background application you're referring to. If I have an app that reads and writes to a bunch of CSV files how is that any different from using libsqlite3 to read and write to an sqlite file?
Imagine if you will that database projects do exist, but don't come with easy to deploy documentation. What you have is stuff in source code tarballs and a vague idea of what dependencies you will need and what versions. You can't just `apt-get install mysql-server` and be done. You have to compile the stuff from scratch. Download a tarball, run `./configure && make`, wait ten minutes, come back to see that some dependency wasn't met, download that, `./configure && make` that, find yet another dependency. And this in some cases is the superior method to the RPM hell you might encounter. Oh and don't forget to supply all the necessary configuration flags to get the features you need, or else you won't be able to use the thing you just installed.
If all you want to do is run a small project, or a big one where performance matters and you have deadlines, you might just "do it by hand" and simply create your own pseudo-database out of flat files, memory dumps, whatever.
I meant RDBMSes that are not SQLite. For MySQL or Postgres or whatever you usually need to do a system-wide package install and have a daemon running in the background. Portability of (compiled version of) your program disappears. That's why desktop apps - games in particular - tend not to use any database other than SQLite or flat file structure.
Were oracle/DB2 systems considered viable options back in the day? Obviously not for a small project, but a company like origin was quite successful then and would have had the capital to buy licenses.
Still tremendously expensive, and not as performant as keeping everything in memory.
The approach described is what people nowadays would call memcache, but the entire architecture is different because client connections are persistent rather than resetting every page load like the web. They just built a standard game server keeping everything in memory in the regular data structures. Fork-and-persist is a nifty way of using the operating system's copy-on-write functionality to checkpoint state.
Their initial strategy involved purposely losing an hour of player data every day. Everyone used to troll newbies by pretending to be nice and giving them free stuff around 11am, and then by 12pm it would suddenly all disappear and revert back to the original owner.
As mentioned in some of the replies to the Quora question, this was also a great time to mess around in PvP and put on all of your best equipment (armor of invulnerability, and weapons of vanquishing) and head out to the dungeons to cause chaos. Or taking 80+ purple explosion potions with you and hucking them at groups of people still fighting in the dungeons. If you died, oh well, like you said everything that happened in that hour got reverted.
Oh the days of the Chesapeake shuffle, what many us used to call the lag spikes and rollbacks the afflicted the game early on. You could tell your servers cycle time with little effort and there were days when everything was just out of sync and even movement from certain areas was not a certain event.
This model apparently allowed for quite a few dupe bugs, one of which I published on an old fan site. It was quite involved, in fact I wrote it up so deliberately convoluted that what you were doing was obvious. Did it work, I wasn't quite sure but I did suffer both duplication of items and loss from their system through no effort on my part.
Its kind of funny to learn how they did it this late, reminds me of mechanisms we employed with some early multiplayer door games on BBSes. Supporting multiple players for some early titles was a bear, the least of which was the lack of stability of most connections just plain bad code that bit you in the butt more often than not.
This website (as far as I know, I don't know if they changed that) since the beginning uses flat files instead of a database. Depending on the case it might actually work and be a valid strategy for persistency
I also think flat file storage is a valid solution. If you are abke to use advanced filesystem features like snapshots and online replications, the use-cases for databases shrink even further.
I built a content pipeline and CMS on top of flat files (json) and mercurial.
You can get repository level transactions by committing on each consistent state (squashing before push) this lets you do undo/redo and recover from failed operations easily - databases sort of give you this - but only for the stuff they actually contain (so unless you're putting your asset files in to the DB you don't get this).
Yeah I have no doubt that flat file are a valid solution, I think the only headache was related to a dupe bug.
"Server-boundary crossing edge-cases and race conditions persisted for a long time -- allowing for gold and items to be duplicated, though I think we had largely eradicated the big ones by the time I had moved on to UO2."
HN itself uses flat files for each of these comments!
Somewhat off topic, but one the pieces of code I'm most proud of was effectively a recursive XML scraper that resolved each leaf (each node was a web request) into a line (like a file path) of a massive text file, which was then grepped when a user searched for something. I hacked it together really quickly to solve a problem, which is solved orders of magnitude faster than commercial solutions that tried to be overly clever. I like it because while it was gross and bad and would be embarrassing to read, in reality grep is crazy fast and the OS cached the text file once you'd read it once. It's a nice personal reminder to me that you don't always need to build some Grand Design, especially when you're just building a tool that has no future evolving path.
MUDs, the both spiritual and technological precursors to UO also used flat files for storage of both maps and users. it's not too surprising the UO storage was the same.
I brought this up on Twitter, but I think the term "shard" referring to a partitioned database originated from Ultima Online.
The game's fiction included reference to shards of a wizard's crystal that created duplicate worlds -- a convenient explanation within that universe for the need to run multiple, duplicate servers with different users on them.
"The word "shard" in a database context may be derived from the 1997 MMORPG Ultima Online, which referred to its game servers and their associated databases as "shards." This was in reference to the game's fiction, in which the broken shards of the evil wizard Mondain's Gem of Immortality each contained a complete, living copy of the world.[24] This usage spread informally through the video game industry, and may have come into mainstream usage via Flickr, which began as a tool for an unpublished MMO."
The only folks I personally know who played UO were my actual friends :) I have coworkers who were also old enough to remember it but they all started withy EQ or WoW.
In my mind no game will surpass closed beta era UO. It had the sandbox and freedom combined with q culture that was actually into the idea instead of the grieving
This is really awesome! I had always just assumed the servers ran on Windows NT or similar. I'm not really sure why I thought that. Maybe because SphereServer and RunUO are/were for Windows so the association stuck in my head? But I never would have guessed Solaris.
It's really hard to have a discussion about UO without the urge to talk about how important it was to me (I've noticed I'm not the only one).
So rather than resist that urge... Ultima Online is directly responsible for me being a programmer. I was 13 when I got a copy of the game and an account thanks to my parents. It wasn't the first game I'd played where I'd wondered how it had been made but it was the first game I'd played where I couldn't even conceive of how it could possibly work. It was so different from anything else that I'd played before, including the first 8 Ultima games. Trying to figure that out led me towards programming in general and to SphereServer, the UO server emulator. The scripting system for SphereServer allowed me to change the game (for me and my friends only, but still) and really explore what's possible. By the time I was 16 I knew what I was doing for the rest of my life, even if it wasn't necessarily anything related to games.
No gaming experience has ever been as involving as the good old UO. And I also started my programming by coding automation scripts for one of at that time popular and mostly prohibited playing assist programs called "UO Inject". No other motivation keeps you as engaged in programming as writing scripts that automatically cut the trees for you while you sleep!
One of the very first actually useful programs I wrote was a simple macro program for UO. It would read a text file, parse the very basic commands in the file and then send them to the UO client by finding its HWND via FindWindow and calling SendMessage to send the key presses. I used this to macro easy skills like hiding with scripts like (I don't remember the exact command names now):
STARTLOOP
KEYPRESS F1 // client macro key for hiding skill
WAIT 1000
MOVERANDOM 4 // move 4 times in a random direction to get around server checks for using a skill in the same spot
You wanted to move 8, not 4. Skill gains were made through an 8x8 grid. All the latter skill macros through RunUO were set up for being done on a boat sailing up and down to the east of Moonglow (where you could wrap around the map). They spammed the skill until you got a skill gain, then halted the boat, stepped it forward 8, and tried agian. Rinse and repeat until the 'run' stopped, then resume just moving forward trying until you hit the next run.
Before the graphics, there were thousands of MUDs. In fact, UO took PvP inspiration from a (still active) MUD called MUME [0], which I have played for over twenty years, and met my wife through. Not many people know about that, but you can find quotes out there by the creators of UO about it. Anyways, MUME doesn't allow scripting, but for me, it was MajorMUD back in the early 90s that started me on the scripting path. Good stuff!
I used to write scripts to chop down trees, practice healing, mine and smelt ore, cast "fireball" on myself so I could level up my magic skills...lol
I remember a specific time when I was setting up my mining operation. I took the time to figure out how long it took for ore to re-spawn, and had my character mine in an elaborate circuit several times before starting the smelting process. I was super proud of myself, haha. I think I was about 12 or 13 at the time.
I had a similar experience. Between the ages of 14-16 I tinkered on my own sphere shard with a friend of mine. I didn't realise what I was doing was programming until I took courses 16-17 and realised what I had been doing the whole time :)
There was one website, Taran's Scripting For Dummies, that was pretty much my bible.
Nice. Ultima 'offline' is directly responsible for me being a programmer. Inspired to build ultima-like games in high school (w00 turbo pascal!), I eventually ended up as a professional software thingamajiggy.
...I'm still sad about how awful the ultima (offline) series ended, though
Just yesterday I was reminiscing the stability of Sun Sparc servers. I recalled the time we moved offices in 1999, and we had to shut down them down. One last peek at the uptime showed they'd been up and running for 6 years or so (turned on when we moved in, turned off when we moved out).
I seem to recall it was a SparcStation 10. No clue about the clock speedm but I believe it was more. Note: my first PC was a 386/25MHz, which was considered blazingly fast (and a cool 10k including the Nec MultiSync 3D monitor, a 256 color EGA video card and 24 bit P2200 printer).
PS A different topic altogether, but for quite a while it seemed that whatever clock speeds we gained was wasted on pushing pixels around on the screen...
> Each shard (the term sharding probably originated with UO)
Probably, as the term "shard" was part of the backstory: a gem that contained the world ("Sosaria") was shattered, each shard then contained a version of Sosaria.
I met with Raph IRL once - he is a really smart guy. He is now working on a somewhat of a spiritual successor to UO called Crowfall. http://crowfall.com
Yeah I am kickstarter funder, I can't wait to finally play this game. Hopefully it will last more than 3 months, I can hardly remember the last MMO I played for more than a year.
As someone has said, the quality of your resume in the game industry (or film industry, I'm told) is only as good as the last game you worked on. Ultima Online was an attention getter for employers for many years; to a certain extent still is. That's unlike most games that are forgotten after a year or two: there are still a lot of old UO fans whose brains were pumped full of dopamine by this thing and will not let them forget it.
On the other hand, UO burned me on game development worse than any other game I ever worked on (though I have kept working on games since then). It was a miserable project. So the effect on the resume was good; the effect on the soul was awful. I never worked on another MMO again and never, ever would.
Can you elaborate on the experience?
I assume the burn out comes from the time constraint.
What would be the antithesis to the burnout you experienced working on an MMO?
Hm, there's a lot to it. Maybe the best way to do this is a Choose-Your-Own-Adventure style of answer.
Do you want to know about...
1) the incredible complexity of MMO development and why nobody wants to make an MMO (though everybody thinks they do)?
2) the origin of Ultima Online as a small, skunkworks project that led to extremely awkward core engineering?
3) the even more awkward munging together of the Ultima IX and Ultima Online teams that led to a suddenly massive team trying to ship a massive game in six months (and succeeding!)?
Ultima Online had begun as a small band of junior developers within Origin back in like 1995. The original concept was to build a massively multiplayer game using a variant of the Ultima 6 engine. That was a pretty far throw back considering that Ultima 7 and 8 had shipped with much more advanced features and Ultima 9—fully 3D and "modern"—was on its way. The group of programmers who worked on Ultima Online for the first few years were brilliant, talented, and driven, and they produced a lot of great work. The game would, of course, never have happened without them. But, they were absolutely green programmers, fresh out of high school in some cases, and... well, let's just say that the Ultima Online engine began to sag under the weight of its massiveness by the end of 1996.
One example just to illustrate the idea. In most game engines even at that time there was a conceptual difference between The Camera and The Player. Though the camera would often be attached to the player, you could normally move the camera someplace else—to show an in-game cutscene in which the player wasn't involved, say. But in the Ultima Online engine this distinction was lost. The camera was the player: where the player was, that's what you were looking at. It meant that when we wanted to show a cutscene, or a clairvoyance effect or suchlike, we had to use a hack: make the player invisible and non-colliding, teleport the player to the scene, move it around like a camera, then teleport it back to its original position and reset its usual state.
This sort of thing was commonplace. Imagine this times 100,000.
So here we have several young, gifted, powerful programmers making history—as they know, though hardly anyone else did—building a brilliant but difficult concept into a working implementation, often naively, sometimes shoddily, but effectively, productively.
Late in 1996 Electronic Arts (who owned Origin) woke up one day and realized that they had a massive potential hit on their hands. A game they could sell not just once, but once per player per month. Game subscriptions were a whole new concept, and boy what a tasty one.
After waking, the next thing EA did was to demand that the game ship by summer 1997. That gave Origin about six months to ship a huge, complex, wildly innovative concept that was barely out of the prototyping phase. This was, to say the least, an ambitious goal.
So they threw bodies at it. The Ultima IX team had been working on "the sequel of sequels" for quite a few years. The team was large but progress was slow. Although Richard Garriott would never let the game die, EA had no real enthusiasm for it (and indeed in the end it shipped, and was quite good fun, but buggy and didn't sell well).
EA put Ultima IX on hold and moved all the developers onto the UO team. That more or less quadrupled the size of the UO team overnight.
So now you have a small, brilliant, fast, naive cadre of programmers for whom UO has been their baby for a few years squished together with a large, perhaps less brilliant, slower, but wiser and more experienced pile of developers who are pissed about losing the project they've waited their whole to complete and find the idea of shipping any game, much less the world's first bona fide MMO, in six months positively laughable.
There was fear. There was anger. There were frenzies of productivity. There were frenzies of laziness. There was hope. There was dread. There was culture shock. There were absolutely awful tools. There were absolutely indecipherable codebases spanning Windows and Unix, with no documentation and the only people who understood it were too busy (or just didn't care) to answer questions. There was a game that could never live, could never run, could never be tested or debugged on a single system—a game that was intrinsically an MMO and needed a shard to run.
There were tiny bugs that took many, many, many weeks to find and fix.
This feels like the right time to quote Dickens: "It was the best of times, it was the worst of times." Only it wasn't. It was just the worst of times. And despite all the joy that UO seems to have given other people, it was a loathsome development experience—I can't think of a worse or more accurate word than loathsome—and after it shipped in September (was it?) of 1997, just about on time, miraculously quickly, I never played it again.
After reading that I feel the need to thank you. Seriously. Thank you so much. As I've said elsewhere in this thread, and every time I see UO brought up, UO is responsible for the entire trajectory my life took since I was 13 years old. I wouldn't have the life I have today if it wasn't for what that game inspired me to do and to learn. Your loathsome experience didn't just bring joy to hundreds of thousands of people it sparked the fire of imagination for at least one person and I'm sure countless others. Someone pays me to program and to solve hard problems today because of the work of everyone who had a hand in UO.
I'd be very interested in anything else you have to say on the topic. As, I assume, would most of the rest of HN. What games did you work on after? Was it something specific to MMOs or UO in particular?
First of all, this was 1997 +/- a year. They amount of stuff you could pull off the internet was radically smaller than today. The number of solved problems was radically smaller. So basically they went were few had gone before and created something successful that persists to this day. So heir career would be looking pretty damn good I would say.
Second, tech stack != career. I'm seeing this sentiment more and more on HN and it's weird. Especially weird in an industry know for its rapid change.
Your question is so backwards it almost feels like trolling.
The "tech stack != career" is on my mind lately since the "stack" terminology seems to be really in vogue these days. Like, I hadn't even heard of a "full stack engineer" until a few years ago and now it's everywhere - which I think is because it ultimately amounts to "general-purpose programmer" and that's actually what most businesses need most of the time.
Everyone(who's been around long enough) knows that the technology churns and real progress is slow, but I think this kind of fashion is based on employee marketability - nobody wants to be just a code monkey, after all.
Something a friend of mine said that really resonates with me, and helps to put the whole thing into perspective: s/stack/pile/
This works splendidly because even the hippest startups working out of mansions by the beach (or whatever) are still fairly kludged together from an engineering standpoint, especially if there is ageism serving as a kind of experience cap (for example ageism by proxy of being too expensive for those startups to pay for). People are extremely territorial about their piles, but if you are good enough friends with those engineers (or get them drunk enough), you find out that everybody lacks confidence in their work to some degree. Someone I work with currently said that if you don't look at code you wrote three months ago and feel like it's shit code, you aren't improving. That is a very insightful comment, to me.
Could you elaborate a bit? Do you mean that you think that's too specialized to go anywhere else? But a solid base of "client-server" abilities at that level of demand will get you in the door in all kinds of places. With a pedigree like that one would merely need not get too drunk before the interview to walk on to almost any "cloud" team.
Eh! I'm a bit more jaded. I've been fed the line "you have awesome background and we wish we could take advantage of it, but your experience with the skills we're looking for is a little too long ago" before.
Well, sure it's all totally new to you and amazing. You're like, what, 900 years old canonically? We don't all get to escape to a parallel dimension where time moves more slowly but we continue to age in our original time line. /topical
Pretty damn good, especially after the turn of the millenium.
Hell I'd hire any of the guys now if that was in their resume somewhere, because that was some nifty stuff. Gotta respect those sort of individuals who show real front-to-back understanding of what it takes to build/implement something people actually use, do it well, and adapt to change while scaling.
I would imagine specialized: resumes matter less and know-how (which is technically covered under most anti-competitive clauses, though IANAL) matters much, much more.
I lucked out and was accepted as a public alpha (and later beta) tester for Ultima Online.
I remember it being very laggy and a bit glitchy, despite having a high-end PC (50mhz 80486, 28.8k modem). But I was still mesmerized with the concept of an MMO, that I could see and talk to and fight with people from anywhere in the world.
So if I understand this correctly the game state was kept in memory in variables/data structures and then once in a while saved down to the DB. Why not just save stuff as they happen? Seem like the most obvious solution to avoid rollbacks.
Do you save write-through or write-back? If you save write-through your game is blocked on your disk write speed, which is not very high, and if you save write-back then (unless you're very careful with journalling) you run the risk of inconsistent state.
(Remember, this was 15-20 years of Moore's law ago so price/performance was about a thousand times worse than today)
I was there on the UO team. Fun fact: this server closet also gained an optical scanner as its security system, due I think to the generosity/love-for-toys of Richard Garriott. The optical scanner didn't work very well, and was way overkill, but it was super neat. We all felt like spies. Despite actually being geeks.
"As Raph also notes, there were no databases originally involved in the storage of game state or player data for UO (disregarding analytics here), everything was kept in flat files. Backups worked by flagging a moment in time where no one was allowed to cross server-boundaries -- during that moment, each areaserver was commanded to fork(), essentially duplicating itself in memory (it's more complicated than this, thanks to Copy-on-Write, but let's simplify). After everyone had fork()ed, the "lock" preventing boundary-crossing was cleared. Then each areaserv began to dump out its huge chunk of memory-state into a file on an NFS server. Those files were then all tarred together and kept as a "backup" of the state of the server. These heavyweight backups happened at half-hour intervals, I believe."