Add one to the list of "people who should never pay for another beer in their lifetime":
In order to extract these programs from most of the DSPs, we had to decap the chips, scan them in with an electron microscope, and 'tweak' the processors to allow us to dump their protected program ROMs.
As you can imagine, this kind of work costs a substantial amount of money. Professional firms that do this can charge upwards of $100,000 per processor for this kind of work.
Thankfully, Lord Nightmare from the MAME/MESS project was in contact with a savant named Dr. Decapitator, with the necessary knowledge and hardware to make this possible. Even better, he was willing to do it just for the cost of the donor cartridges and supplies. This worked out to $250 per coprocessor.
"Yet once again, we were stuck looking at a binary blob. And this time, we had no real HLE code to go off of. Luckily, Cydrak recognized the ISA just from looking at the binary: it was an ARMv3 CPU!"
From the perspective of a software guy, this sounds very impressive. Can some hardware guy chime in and tell us just how impressive it is to recognize a CPU from looking at a binary dump?
The instruction size is the first thing that is apparent. By looking at the pattern you can distinguish between fixed-sized, or variable-sized instructions, telling a lot about the architecture. ARM either has 16 bit (thumb mode) or 32 bit (normal) fixed-size instructions.
Also for every ISA there are some instructions that are very common, for example x86 has very-recognizable NOPs (0x90).
Having said that, seeing it is ARMv3 code specifically in one glance is pretty impressive.
ARM is by far the easiest ISA to do this with: since the vast majority of instructions begins with an "always" condition code, and the condition code is a nibble (0xe), you can pick out ARM instructions easily.
Man, look at the games that shipped with an SA-1. Its an independent (not slaved to the system CPU) 10mhz C5C816 with its own scratch RAM and can DMA to main system RAM; the CPU of the SNES itself is a 3.58 mhz C5C816 (technically a 5A22, Richoh's clone of the 65C816), and the scratch RAM (something like 256kb of it) is faster than system RAM.
Remember how Super Mario RPG was really expensive when it came out? Thats why. Quite a few other games used this chip, but it seems to be all Japan only except for Kirby Super Star and Kirby's Dream Land 3, neither of which were particularly exceptional games.
And yet for some reason gamers complain that games of 2012 are too expensive. $60 of 2012USD really isn't that much compared to $50 of 1992USD. Add in the fact that you can get some excellent an amazing games on mobile platforms now for a dollar, and gaming is cheaper than ever.
I wish there was a good data source on that, because it would be interesting to see.
I'm not sure if the single player play time has really gone up or down to get through the main story, but modern games seem to incorporate almost endless numbers of side quests, self generating quests, achievement junky-only quests (kill 10,000 of X) and also a lot of multiplayer functionality.
I love how game consoles are always providing exciting hacks - from the variety of CPUs and DSPs used in SNES cartridges to the brute-forced SHA collision exploited to hack the PSP.
For what it's worth, this is almost identical to the process initially used to hack satellite TV smartcards, except it seems that these devices had a lot less physical tamper protection (and can't be reprogrammed with a new ROM over-the-air).
I think this kind of console hack is a dying art - as process shrinks and most consoles move closer and closer to commodity hardware and off-the-shelf software, LLE and chip decapping become less and less appealing and software exploits are the name of the game.
Absolutely awesome, though - this is the kind of story I come to HN to read.
Documentation is great in that it can elaborate in much more detail, especially with actual pictures and graphs.
However, unlike source code, documentation cannot be machine tested for correctness.
Honestly, they're both important, and for different reasons. In this instance, Cydrak and I were able to pass revisions of the ARM core back and forth to fully debug it.
No doubt books are very useful learning tools. However, depending on the code, source code can be equally if not more useful. For example, years ago I learned a lot about C by reading (and later writing) Linux kernel source code.
It's also true that code being open source doesn't mean the code is useful, but a book being published doesn't mean the material is presented well, either.
Maybe it's just my pro-open source bias, but it feels like you've got an axe to grind against open source. If so, would you mind sharing why (maybe you know something I don't)?
I'm sure you weren't implying me in particular, but to clarify, I certainly don't think open source is the solution to everything.
I just find it particularly important in the field of emulation. This is much like a science, with constant refinements to theories. Since we're running against the clock to preserve these systems before they die, I assign much more value to open source emulators than I do to other applications (for instance, I'm happy to use the closed source nVidia driver on Linux.)
This is fascinating, but I'm not really clear on what role the electron microscope played. Did it just allow them to see where they needed to modify the chips to allow them to dump their ROM?
That's correct. The data was also extracted using probes under a microsope in this case.
From Dr Decapitator's page:
"The data was extracted by laser-cutting the top two ROM bits from the actual CPU logic (so it would read all opcodes as OP(00) or LD(11) opcodes depending on whether lines were pulled high or low, and neither OP nor LD can branch address), then microprobing, 8 bits at a time, the ROM outputs from before the cut. The program counter was also monitored to ensure that the data read started at 0×000 and didn’t jump anywhere."
Oh, good. I was intentionally vague there, as I wasn't sure how much was public knowledge.
Yes, basically we bypassed the JP (jump) and RT (return) opcodes, so that the program counter always stepped by one. Externally clock the CPU to move to the next instruction, and keep logging the data lines prior to the OP/LD cuts.
Can any MAME/MESS hackers tell us if we'll be seeing accuracy improvements now that this information is available. MESS is my console emulator of choice.
I wrote the g65816 and spc700 cores used in MAME/MESS, but I haven't been involved in the emu scene for almost a decade now. I could take a look and see if there are any quick things I could graft in, but my time is so limited these days...
I wouldn't be too surprised if the details from all of this get pulled into MAME/MESS. It will take some rewriting though as I believe BSNES is written using a number of C++ features (I seem to recall it even needing some C++11 stuff from when I last played with the source). The CPU cores themselves I'd imagine would be fairly easy to port over but I'm not sure how tied in everything is for timing and everything else in there.
The main magic I do for RISC CPUs is allowing a write callback binding to the registers (overloading operator=) This allows us to handle pipeline invalidation on writes to the PC register, without having to special case the hell out of every single instruction that can modify it.
Ah seems my memory was bad there. I must have had it confused with another emulator, however that sounds while it wouldn't be trivial to rewrite those sections it shouldn't be very difficult to do so either, just very very tedious. I think other than that there could only be a licensing issue, and I seem to recall that bsnes had a fairly liberal license that shouldn't run into any problems. Though obviously you'll know better than me byuu :) In that case it's all about time and who has it that'll determine when/if it'll happen
In order to extract these programs from most of the DSPs, we had to decap the chips, scan them in with an electron microscope, and 'tweak' the processors to allow us to dump their protected program ROMs.
As you can imagine, this kind of work costs a substantial amount of money. Professional firms that do this can charge upwards of $100,000 per processor for this kind of work.
Thankfully, Lord Nightmare from the MAME/MESS project was in contact with a savant named Dr. Decapitator, with the necessary knowledge and hardware to make this possible. Even better, he was willing to do it just for the cost of the donor cartridges and supplies. This worked out to $250 per coprocessor.