As someone who has only ever written about 20 instructions worth of assembly, I am kind of torn.
The kind of cleverness needed to thwart the decompiler's efforts is very impressive. A part of me wishes I had been around at a time when assembly was an acceptable "language" instead of the last resort it is today for most scenarios. Getting this close to the metal (or silicon) and being able to pull off such a stunt must be exhilarating.
At the same time, another part of me is very happy I do not have to deal with such low-level details. Debugging code written in assembly (with assembly being the main language, not just inline assembly in a C/C++ program or something like that) must have been exquisite torture.
One can also claim that today’s programming is torture.
Back in the microcomputer days, all you needed to write a program was a desk with a single computer, one or two tools that you knew through and through and a few reference books.
Nowadays, we sometimes spend days or even weeks choosing our tools and libraries of sometimes questionable quality and getting them to work together before we can even start thinking of the problem we want to solve, and even then, we still spend half our days googling, and being lucky if we get semi decent answers.
My most formative years as a programmer mostly involved 68040 assembly on the pre-PowerPC Macs. Back then, I used CodeWarrior a lot, and if there was a head-scratching bug in my software (as there often was), I could launch it from a fully-featured debugger, set breakpoints, skip ahead to a block of code, step through it line by line, see every single value that existed in raw hex on the heap, see exactly what was on the stack. If I was wanting to flip bits in someone else's software, I had MacsBug, which was a stop-the-world OS-level debugger that did all the same, but even better. [1] Back then, there was a very definite sense that I owned the computer, that it was my tool, that it would do precisely and exactly what I wanted it to. In a way, whether something was open source or not didn't matter; I could modify it anyway, and often for less effort than it takes to crawl through a modern over-architected codebase.
However, now I can glue some really impressive libraries together, written by other people and available on massive centralized code repositories, and deliver an application to as many people as I want, all running different hardware or operating systems (within reason) but still all getting the same software experience (...mostly...), and even charge some money for it without having to worry about somebody cracking the license check on my shareware.
The effort required to make complicated new software has gone way down. The effort required to troubleshoot and fix complicated software has gone way up. Nowadays, when I'm doing the former, I'm happy, and when I'm doing the latter, I'm not.
I think that might be part of why there's so much churn in the software industry today.
[1]: MacsBug did this in real-time. Start up an application, drop into MacsBug, set a breakpoint on a function call, switch back to the OS and application, trigger the function call, then step through the code in MacsBug. If you were careful, you could even "rewind" code under certain conditions, so you could, say, change a branching opcode, back up, go back to the OS and see if the change did what you want.
As someone who started with Z80 and 6502 assembler, I think that in some ways the old days of programming were better, as there were just fewer libraries and APIs. You were working close to a bare-metal machine.
These days you can get a lot more done, more easily too, and help is easier to find, but there are a lot more levels of libraries between you and the machine. Choosing the correct platform takes time and sometimes guesswork.
Debugging was just different. You had to get into the mindset of the original developer and understand what he intended and the impact of poking a memory-mapped location (could cause a graphics page flip, or toggle the speaker, or executing something in ROM). Comments were your friend. Without, yes, it could be a challenge.
But decompiling & debugging wasn't just limited to ASM. I once worked for a shop that lost its original C source code and only had a copy of the executable. (major source code control failure - no backups of the server). They decompiled it and worked from the resultant C code. It was as bad as you can imagine, and enormous amounts of time were spent renaming "variable1" to "customerName", or methods from "sub232" to "CalculateTax()", etc. After a few months the code was sprinkled with meaningful names, but it was still very difficult to read and felt very unnatural, formal, and distorted. It also ran slower than the original when recompiled.
But as for defeating a disassembler, I recall a few strategies stretching back to the late 70s/early 1980s on the Apple ][, when the "disassembler" was a human being rather than a tool. This is probably more appropriately called obfuscation or even copy protection, but it feels related. There were a variety of approaches taken:
* Self-modifying code. This was the most common approach. Code would loop through areas of memory, incrementing/decrementing values by set values or based on an algorithm to produce valid opcodes. At any point in time, much of the code loaded was nonsense, so debugging had to be done in decoded blocks.
* Jump tables (and indexed jump tables with offsets calculated at runtime) so that the code would skip all over the place. This was mostly just an irritation, rather than a hard obfuscation.
* Burying executable code in resources (sounds/images) and related techniques such as code that calculated its hashvalue and used that to self-modify to 'unlock' executable blocks.
*Overlays. Code was loaded on top of unused resources, and resources loaded on top of code.
And while it wasn't a decompiler defeat, per se, changing the timing for the floppy disk drivers which read from non-standard disks would prevent one from simply loading data from disk and decompiling -- only that code that is in memory could be examined. Nybble-copiers were created to defeat this approach.
One or all of these approaches could be combined to produce code that was a nightmare to debug, but ultimately anything that the computer could execute could be decompiled, eventually.
For the Apple ][ there was even a card (the “wildcard”) which would interrupt the CPU and dump the memory to disk along with a special loader, allowing you to reboot from the disk and resume where the CPU left off. I don't think it had any valid use besides defeating copy protection:
There is a great collection on the internet archive where someone has gone through and unprotected tons of apple][ disks and documented the copy protection. It's pretty good reading, but a little repetitive after a while.
https://archive.org/details/apple_ii_library_4am&tab=about
Being able to use hardware tricks for programming, like non-standard display modes, self-modifying code, or fit a few more bytes into 64 KB was worthy of geek brownie points, specially at demoscene parties.
>A part of me wishes I had been around at a time when assembly was an acceptable "language"...
Probably not as glamorous as you're imagining. It took a long time to get anything done, and most of that time was spent finding bugs. All the time a lot of people thought the real cool people where the ones before who had to program using wires and vacuum tubes.
Best advice is, don't try to mimic those of the past, become a master of your own domain and change with the times.
I'm surprised the Movfuscator hasn't been mentioned yet. It compiles C code into unconditional MOVs, and if you watch the author's Derbycon 2015 video there are ways you can scramble the MOV instructions, truely making it a decompilation nightmare.
This is because "movfuscation" isn't a practical option for people actually trying to ship binaries that still perform well for customers but resist reverse-engineering. One of the battlegrounds for this sort of thing is the tug of war between game developers and cheat developers, and games still need to perform very well. Things the author mentioned, like address-rewriting at runtime, don't incur a performance penalty.
Just rebuild the sensitive portions of your code using movfuscator and leave the performance critical stuff alone. As long as everything is statically linked and you don't do anything stupid like "if (check) unlock()" that can be easily patched, it would make life pretty miserable for the RE crowd.
I don't know if decompilers are able to do something about it, but there's a "neat" technique where your machine code can be interpreted as two different sequences of instructions depending where you start instruction decoding. For a simple but artificial example of what I mean:
The following sequence of bytes:
b8 50 83 ec 10
Decodes as:
mov $0x10ec8350, %eax
if you start at the first byte, and, if you start at the second byte, as:
push %eax
sub $0x10, %esp
Here I essentially hid an instruction in the mov'ed data, because that's the easiest way to create something like that, but I've seen mind blowing examples of this technique. I unfortunately don't remember where.
That's an issue for static disassembly, which is step one of decompilation. Your snippet is a classic example that can throw off naive linear-scan disassemblers. Recursive disassemblers can handle it easily if (and it's a big if) they can identify the basic block entry points. If you have top-level entry points (which can also be a problem), the main problem for a recursive disassembler are indirect jumps that don't fit standard patterns like switch jump tables.
All of this gets easier if you can augment your static analysis with control flow traces from program executions with coverage of the relevant branches, so you don't miss basic block entry points.
> All of this gets easier if you can augment your static analysis with control flow traces from program executions with coverage of the relevant branches, so you don't miss basic block entry points.
That's only true if the traces you can generate do a reasonable job at covering the states the application can find itself in, which is a big assumption.
Yes, that's what I meant by coverage of the relevant branches. But the good news is that all these partial, heuristic sources of information can be combined. E.g. for detecting function entry points, you can combine information from ELF/PE export tables (if present), function prologues detected by a linear scan, vtables, static CALL targets, dynamic CALL targets from run-time traces, etc.
This is a really cool post. You can also target IDA itself directly, rather than the decompiler, making it difficult to even view the disassembly. It's been awhile since I did any of this kind of work (it's relevant to software security tokens, games, and content protection), but within the last few years people have published IDA RCE memory corruption, so I imagine it hasn't gotten too much harder to hopelessly confuse IDA.
There's an interesting vein of research work here in making software reverse engineering more difficult, and measuring how much more difficult.
A precursor to decompilation is control flow analysis, the production of the control flow graph you see in the "before" stages in all of the examples in this post. You can go one step further, on a good day, and make it very difficult (perhaps very very difficult, perhaps impossible) to recover a precise control flow graph for a function.
There are a few different ways to do this, and I like these approaches more than targeting specific heuristics in IDA/Hexrays because, on a good day for the obfuscator, you can make a theoretical statement about the work effort required to un-do the obfuscation. If you can make that work effort large, then you start to have a security guarantee that is a shade of the security guarantee you get in cryptography. The methods outlined in the parent blog post are great because you can start using them today, but if they annoy Ilfak enough, he'll fix them and they'll stop working.
There are also undocumented instructions and prefixes. For example at this year’s DEFCON someone presented a particular kind of e9 jump prefixed with 66. That instruction is incorrectly disassmbled by all hitherto known disassemblers (IDA, gdb, objdump, VS, etc). And since that will change the length of the instruction, effectively you can make the disassembler produce garbage.
The techniques described in that document are different, but in a way they are alike - both use dirty tricks to shield themselves from people trying to reverse engineer them. The only difference is that one takes place "at compile time", in way, while the other works in a dynamic program as it is executing.
It's been long time since I've hacked anything but first one had fun by storing the decompression code into the interrupt table (yes, there). Or by making sure the decompression routine used accurate timings (so that if you fiddle with step by step execution it fails). Or calculating jump's based on values on the screen (that is, not the values show by your debugger).
That way the decompiler should, at least, be able to execute the code.
But that was before the protected mode in intel. My knowledge stops there.
The kind of cleverness needed to thwart the decompiler's efforts is very impressive. A part of me wishes I had been around at a time when assembly was an acceptable "language" instead of the last resort it is today for most scenarios. Getting this close to the metal (or silicon) and being able to pull off such a stunt must be exhilarating.
At the same time, another part of me is very happy I do not have to deal with such low-level details. Debugging code written in assembly (with assembly being the main language, not just inline assembly in a C/C++ program or something like that) must have been exquisite torture.