(works better with accessibility tools, like text to speech; and translators)
There are also new PDF and EPUB editions here: https://www.patreon.com/posts/67035520 (it's a public post, I just needed a place to upload those files without consuming my hosting's bandwidth).
One obvious thing you could do to conserve bandwidth is not do loading=auto but loading=lazy. You've got like 70 images in there which are going to download, but how many readers will scroll to the bottom and look at all 70? (Few.)
Super minor nitpick but ROP wasn’t used for the King Kong/SMC exploit. I would consider it a ROP exploit if multiple return-oriented gadgets were chained together to form a full exploit chain, but what happened here is the syscall handler was invoked with a malformed index causing a single jump to user-mode code with kernel-mode privileges. It’s not too dissimilar to calling an arbitrary function pointer.
Otherwise this is a great comprehensive rundown!
I was just recently talking to a coworker too about how I think the Xbox 360 was the first consumer device to have the following types of attacks done to it:
1. Hypervisor attack to then reboot the console into a newer system OS version to retain the vulnerable hypervisor but be able to play new games and get online. This required soldering a separate flash chip to hold the newer system files.
2. Fault injection (reset glitch hack) to attack the system's bootloader
As a teenager who learned programming/hacking by messing with the Xbox 360, I'm thankful that Nintendo is still keeping the dream alive for our next generation of hackers.
I used to have a G5 Mac living a second life as a doorstep with a “property of Microsoft” sticker on it because they were used as early alpha devkits for the 360. Given away at the company I worked for a year or two after the 360 shipped. Rumour has it they were even delivered at the dead of night so no one would see the MS branding on Apple hardware.
I wish I could remember enough to contribute, especially to the DEV/XBL online chapters.
If I ever recover my old /modding/ folder, I will reach out [OP].
I was pseudo-privy to a lot of non-public stuff that I think the public could benefit now from, with a decade passing, I think it would be be appropriate to share.
This thing branched horribly. You had to use instructions to selectively write to distinct memory locations to avoid typical branching because misprediction was expensive.
Here's a good explanation of some of the issues w/ common programming patterns at the time colliding with the 360 design. This was a huge problem with the initial ports of older game engines to the 360 that required a lot of rewrites to achieve ok performance.
On top of being power architecture, both the 360 and the PS3 chose to push the 4 GHz clock limit before everyone else. To do this they sacrificed lots of speculative and out of order execution features of the CPU. The thinking of the hardware engineers was that the software is compiled for a fixed architecture doing a fixed job so the compiler should be able to statically order the instructions to make the best use of the very long, very static, very linear pipeline of the CPU. In practice, that only makes sense for very long stretches of instructions with no branches and no cache misses —which is completely the opposite of every piece of gameplay code ever written. The CPUs were great at large scale linear algebra. Not great at much else.
One thing that Intel and AMD do better than any other player in the industry is branch prediction. An absolutely stupifying amount of die area is dedicated to it on x86. Combining this with massive speculative execution resources and you can get decent ILP even out of code that's ridiculously hostile to ILP.
Our modern CPU cores have hundreds of instructions in flight at any one moment because of the depth of OoO execution they go to. You can only go that deep on OoO if you have the branch prediction accurate enough not to choke it.
> An absolutely stupifying amount of die area is dedicated to it on x86.
Yep. For example, on this die shot of a Skylake-X core,[0] you can see the branch predictor is about the same area as a single vector execution port (about 8% of the non-cache area).
> One thing that Intel and AMD do better than any other player in the industry is branch prediction. An absolutely stupifying amount of die area is dedicated to it on x86.
Zen in particular combines an L1 perceptron and L2 TAGE[0] predictor[1]. TAGE in particular requires an immense amount of silicon, but it has something like 99.7% prediction accuracy, which is... crazy. The perceptron predictor is almost as good: 99.5%.
I wrote a software TAGE predictor, but too bad it didn't perform as well as predicted (heh) by the authors of the paper.
Everything is relative. They do things that seemed quite neat in the 90s, but then progress slowed to a crawl.
I'd call the state of the field quite bad. For example they do embarrassingly little for you to help with the 2 main bottlenecks we've had for a long time: concurrency and data layout optimization. And for even naive model (1 cpu / free memory) there is just so much potentially automateable manual toil in doing semantics based transformations in perf work that it's not even funny.
A large part is using languages that don't support these kinds of optimizations. It's not "C compiler improvements hit a wall", it continues "and we didn't develop & migrate to languages whose semantics allow optimizations". (There's a little of this in the GPU world, but the proprietary infighting there has produced a dev experience and app platform so bad that very few apps outside games venture there)
There's a whole alternative path of processor history not taken in the case that VLIW had panned out, and instead of failing because of optimism about compiler optimizations.
They do a good job but the scheduling aspects are really really fuzzy.
LLVM and GCC both have models of out of order pipelines but other than making sure they don't stall the instruction decoder it's really hazy whether they actually do anything.
The processors themselves are designed around the code the compilers emit and vice versa.
Optimizing compilers aren't that great on x86. Sure, they're good enough to make something 60fps that wasn't before, but they don't really have much specific x86 knowledge.
As said in other comments, not a power thing. Example, the Nintendo Wii U chose to use 3 1.2ghz (iirc) Out of Order execution cpu's that were more like a PC than the cores in the Xbox 360 even though sounding similar in instruction set and composition.
This lead the Wii U to be able to do things like Run Mass Effect 3 and Deus Ex better (arguably) than the PS3 and 360 most of the time. The Wii U was probably the better hardware platform in hindsight but it came too late and the development tools were not as robust so ports just kinda afterthoughts.
You're comparing the 2012 Wii U against 2005/2006 360/PS3. The PPE was indeed terrible but it's not clear that IBM had anything better available at that time.
The Wii U also needed to be backwards-compatible with the Wii, which used the bespoke paired singles and locked cache line features of the GameCube's PPC 750 derivative. This almost certainly locked them out of newer PowerPC designs without more engineering work than Nintendo would be willing to put into its systems.
For context, Nintendo has always been weirdly quirky and low-buck when it comes to core silicon engineering. The Switch is a Tegra X1 in a trenchcoat, the SNES used a 65C816 at about half the clockspeed it needed to be[0] and had half the VRAM removed at the last minute, and the NES stole[1] the 6502 masks so they didn't have to pay MOS for legit chips. All of those design decisions were made purely to improve margins and genuinely constrained game developers in the process. "Lateral thinking with withered technology" is kind of just their thing.
At least now they're 100% on board with a silicon vendor with a sane roadmap, so they'll at least have a steady supply of backwards-compatible last-gen chips to repackage.
[0] At least it wasn't as slow as the Apple IIgs they pulled it from
[1] Technically legal as IC maskwork rights did not exist yet. This is also why decimal mode was removed - it was literally the only thing MOS had a patent on in the 6502 design.
I've never heard anything suggesting that video RAM was removed. AFAIK, the SNES was planned to have only 8KB of main RAM, which was increased to 128KB by release. I think any support for 128KB VRAM was for future proofing, like if the SNES's hardware was reused for arcade systems, or something.
The Genesis's video chip can support 128KB video RAM as well, which besides allowing a larger variety of tiles on screen and doubles DMA bandwidth. It was used in the System C2 arcade board. The Genesis was originally designed to use 64KB video RAM, but after hearing about the SNES, support for 128KB was added. Then they decided that the extra RAM didn't make enough of a difference to justify the cost, so they left it at 64KB.
It wasn't a POWER thing since POWER has been superscalar since the 604 back in '94. This was because when you're designing for a console and paying by the wafer instead of shopping around for a pre-finished unit one needs to consider die area more strictly. Something had to give in the design and given MS controlled the design of the whole stack they thought they could do parallelism at the thread level in the OS scheduler rather than having to devote massive amounts of die area to it.
The xenon core was a slightly modified cell ppe, where there it was focused on saving gates for the cores needed for system management versus more spus.
The idea using it in xenon without the spus was that high perf code could be tailored specifically for this core's uarch being a console, so the in order nature wasn't the worst thing and the gate savings are pretty huge.
I love this series. A minor point about the layout only: I know you are very fond of those tabbed boxes (as I've seen them in every such article of yours that I read over the years), and I bet they make a lot of sense when reading on a mobile device, but on my desktop browser with plenty of screen space I often wish they weren't there. I.e., I wish everything was always visible and maximally require (normal vertical) scrolling.
This is especially true for things you intend to compare. For example, I was awkwardly flipping between the "Xenon" and "Cell" tabs a lot, to compare the two block diagrams.
Modding my JTAGed Xbox 360 to go online and then getting banned and bricking it while trying to unban myself was my first true moment in tinkering, good times :')
Building the little probe used to read the x360 dvd drive key out of discrete components with zero knowledge of electronics was probably one of the most fun technical things I did in my early 20's. It was a logic level shifter if I remember correctly, wired to my PC's serial port.
That 2008 Windows Live Mail screenshot hit me right in the feels :') That's the period of time I got my first "high bandwidth" connection (really anything faster than a 56k dial-up connection), and basically the true start of my programming adventures...
Wow what a journey the homebrew section of that article was. As someone who grew up heavily involved in the evolving Xbox homebrew scene, I dipped out of consoles before the 360 was truly cracked. I remember the sadness in realising the DVD drive exploit had effectively killed off any interest in trying to run homebrew, but I'm thoroughly glad to learn that the community not only persisted, but became incredibly inventive.
I can relate. I had a modded dvd drive. Got banned after a year maybe. Loved playing gears online.
Then I think it got the red ring? That was the end for me. Only recently got into gaming a bit more now. I still like how consoles are mostly set and forget compared to pc based gaming. Looks like the ps4 is getting homebrew which is interesting as it is basically a pc?
I wonder how true this rumor is, but I remember reading that around the time the F35's computer system was designed, Lockheed Martin shopped around for CPUs from different vendors. PowerPC was a very popular architecture used in military hardware, so LM contracted IBM to design a CPU for them.
In Aerospace applications, it's very popular to have triple-redundant system, that's why IBM designed an unusual 3-core PowerPC CPU. Later, when Microsoft came into the picture, they used the design, with essentially minimal modifications.
I haven't followed the F-35s development that closely, so I have no idea what they ended up using, but it's an interesting rumor nontheless.
FWIW, the Xbox 360 chip was based on the processor developed for the PS3. No mention of F-35 here. “These cores are slightly modified versions of the PPE in the Cell processor used on the PlayStation 3” https://en.m.wikipedia.org/wiki/Xenon_(processor)
I finally got through this, coming in and out throughout the work day. I owned 3 360's over the lifespan of the console (a launch version, a replacement after it red ringed a few years later, and then a slim), and I never even brushed up against the cracking or homebrew scene. Very interesting stuff!
I am curious though, other that one brief mention, why didn't you touch on the early hardware reliability issues at all?
Also, either the site has been swarmed and its down right now, or the links in footers 117, 118, and 119 are bad.
> I am curious though, other that one brief mention, why didn't you touch on the early hardware reliability issues at all?
I guess because they likely didn't have anything to do with the architecture? Or were there really reliability issues that were a result of the architecture, instead of physical/electrical hardware issues?
Fair. I was thinking in terms of the physical architecture of the boards, cooling solutions, and entire consoles rather than the pure architecture of the cpu, gpu, etc.
Yeah, it was mainly because the article got too long, so I had no option but to draw the line somewhere. There's still a tiny mention at the footer below the main motherboard image:
> Showing the 'Xenon' revision (the first one), taken from my model from 2005. Xenon motherboards are also famous for being defective by design (they get too hot to play games with!). Remaining GDDR3 chips are found on the back.
Btw, I've replaced the bad links with archived ones. I triggered an archive of that site back in December just in case such great source would disappear, never expected to happen just after releasing the article!
It's funny to me that IBM - a company not known to promote fun and games - ended up providing the processors that powered the PS3, XBox360, and I think the GameCube and Wii, too.
Thank you very much for the deep dive, I imagine the research must have been exhausting (but also rewarding, I hope!).
IBM was the real winner of the seventh generation console war.
Though by not as much of a dramatic margin (Nintendo stayed with IBM for the Wii U and Nvidia for the Switch), one could say AMD won the eighth- and ninth-gen console wars in a similar fashion.
Great read and made me nostalgic. There was also a way to turn a jtag/rgh console in to running dev console stuff. Also the trick where if you moved a dev consoles update file to your dev kit's main directory then it will allow 100% r/w/unsigned code. Modern xbone dev kits require their IP to be allowlisted/brick/turn to a retail if not constantly activated. Though there was the ps4 dev kit trick where you could set the console's clock back to when it was activated to unbrick/unlock it again. I wonder what protections/exploits the ps5 dev kits have.
Working with the ESP32 is what introduced me to the concept of eFuses. Really neat. Allows you to run encrypted firmware images which cannot be easily decrypted even with full physical access to the hardware.
I'm thankful for the people that did the work of reverse engineering/modding the Xbox and Xbox 360, without them I'm not sure if I would have gotten into programming as early as I did, around 13 years old. I got started with using simple modding tools for Halo 2 on Xbox then moved on to writing code for modding COD MW2 on the Xbox 360. Good times, it lead me away from spending all my free time gaming for fun and into coding for fun.
Great resource, nice work putting it all together. I was on the software side of 360 (mostly C++ and Lua), so I didn't get too deep in the hardware. Most of my time was spent modifying and building on the XAM, glad to see it mentioned briefly in there!
The old screenshots of the dash bring back lots of memories. The releases were all named after European cities at the time; Berlin, Geneva, Stockholm, Madrid, etc.
Super exciting read! I was interested in the Xbox 360 modding scene and re-reading this brought back a ton of memories from reading about all this stuff the first time!
Thanks for compiling this, the quality is excellent and the content is approachable.
Hi, this took me 6 months to write, I've also tried as hard as possible to cross check all my references, and include them in the article in a way the reader can find the citations just like in academic writings.
Statements denoting speculation are started with words like 'presumably' or 'it is assumed that'.
I've also took an extra month to send the draft to many experts (part of whom are still active in the Xbox 360 scene) to gather feedback and make all the appropriate corrections. See the 'Changelog' at the end of the article.
I'm afraid I can only do so much. Just like you said, this is all voluntary work. I also keep a repo with all the manuscripts to correct all the mistakes after publishing any content (https://github.com/flipacholas/Architecture-of-consoles).
I don't know how else I could improve this, but I'm always open to suggestions.
I don't know what you mean with reorganization and editing.
You should ask yourself “why am I writing this document“, and then check to see if the document is achieving your goals. I am guessing your goal is “I want to serve as the definitive publicly available textbook for the Xbox 360.“. If that’s your goal, I would split the doc into chapters by topic, and then edit the heck out of each chapter. Look at a model textbook like Computer Architecture, a Quantitative Approach, and see if you can imitate their style.
I think a good technical editor could help you edit this down to about 25% of the words, and still cover the same information.
The job is too big for drive-by code review style comments. You’re going to have to put in the work yourself if you want to improve this.
(works better with accessibility tools, like text to speech; and translators)
There are also new PDF and EPUB editions here: https://www.patreon.com/posts/67035520 (it's a public post, I just needed a place to upload those files without consuming my hosting's bandwidth).
If you spot a mistake, please log an issue on the repo (https://github.com/flipacholas/Architecture-of-consoles) so I can review it. Thanks!