One thing worth noting here is the general simplicity and orthogonality of the addressing. x86 gets a lot of CISC hate for having complicated addressing modes, but if you accept that you're going to have the concept, this is actually pretty clean. You have a clean set of 8 registers in both 8 and 16 bit modes[1], your "pointer" register for a memory operation can be one of the "base" or "index" registers, or a sum of one from each category, and you can have an immediate added.
Writing assembly for the 8086 was actually quite pleasant, the space of stuff you could do was broad and the expression was clean.
[1] Though not the same registers. The list of registers is populated differently for 8 bit and 16 bit instructions. The 386 then played the same trick AGAIN, but placed the selector bit in the code segment descriptor and did a weird dance with how to encode the ESP register (to flag a new "SIB" byte for extended modes), and made a giant hash of things that no one could understand. Writing assembly for the 80386 was definitely not pleasant, though it was actually an easier target for compilers.
The 386 ModR/M isn't that hard to figure out --- it's just that the 8 combinations of addressing modes become [r32], [imm32] is still where [ebp] would otherwise be (0r5), and the [esp] position (xr4) is where they put the SIB.
On the other hand, the 64-bit addressing modes that AMD created with the 64-bit extension are a lot weirder.
The only change done by AMD to the addressing modes was that from two possible encodings accepted by 386 for absolute addressing, the shortest one is redefined to mean PC- (a.k.a. IP-) relative addressing.
The only weird part is that AMD has decided for some very stupid reason to not decode 2 of the 3 bits provided for register numbers in the REX instruction prefix.
Perhaps this AMD decision allowed them to obtain an extra 0.1 GHz in the clock frequency of the 2003 model of Opteron, but it has complicated forever the optimal register allocation on the Intel/AMD CPUs.
While on a 386, with respect to memory addressing (when segmentation is not used) there are only 3 kinds of general-purpose registers (SP, BP and all others), due to this AMD mistake in 64-bit mode there are 5 kinds of general-purpose registers (SP, BP, R12, R13 and all others), where each of the 5 kinds of registers has a slightly different behavior, e.g. resulting in different program sizes, depending on how the registers are allocated.
It's not so much that it's hard to figure out, it's that it's hard for a Regular Jane assembly programmer to know how to do well. There's an embarassment of modes with complicated performance characteristics, and a lot of decisions to be made.
So yeah, serious domain experts and modern optimizers can do really well with the architecture, but it's a huge pain to try to "get stuff done" in, and it's from an era where serious Stuff was still gotten done in hand-coded assembly as often as not.
Basically: with a Z80 or 6502 or 8086, if you had a task "X" to do the "best" way was as often as not the obvious way. With the 386, that was suddenly no longer true and you had to have a bookshelf full of Abrash tomes and whatnot. It was a very different feel.
Writing assembly for the 8086 was actually quite pleasant, the space of stuff you could do was broad and the expression was clean.
[1] Though not the same registers. The list of registers is populated differently for 8 bit and 16 bit instructions. The 386 then played the same trick AGAIN, but placed the selector bit in the code segment descriptor and did a weird dance with how to encode the ESP register (to flag a new "SIB" byte for extended modes), and made a giant hash of things that no one could understand. Writing assembly for the 80386 was definitely not pleasant, though it was actually an easier target for compilers.