The 386 ModR/M isn't that hard to figure out --- it's just that the 8 combinations of addressing modes become [r32], [imm32] is still where [ebp] would otherwise be (0r5), and the [esp] position (xr4) is where they put the SIB.
On the other hand, the 64-bit addressing modes that AMD created with the 64-bit extension are a lot weirder.
The only change done by AMD to the addressing modes was that from two possible encodings accepted by 386 for absolute addressing, the shortest one is redefined to mean PC- (a.k.a. IP-) relative addressing.
The only weird part is that AMD has decided for some very stupid reason to not decode 2 of the 3 bits provided for register numbers in the REX instruction prefix.
Perhaps this AMD decision allowed them to obtain an extra 0.1 GHz in the clock frequency of the 2003 model of Opteron, but it has complicated forever the optimal register allocation on the Intel/AMD CPUs.
While on a 386, with respect to memory addressing (when segmentation is not used) there are only 3 kinds of general-purpose registers (SP, BP and all others), due to this AMD mistake in 64-bit mode there are 5 kinds of general-purpose registers (SP, BP, R12, R13 and all others), where each of the 5 kinds of registers has a slightly different behavior, e.g. resulting in different program sizes, depending on how the registers are allocated.
It's not so much that it's hard to figure out, it's that it's hard for a Regular Jane assembly programmer to know how to do well. There's an embarassment of modes with complicated performance characteristics, and a lot of decisions to be made.
So yeah, serious domain experts and modern optimizers can do really well with the architecture, but it's a huge pain to try to "get stuff done" in, and it's from an era where serious Stuff was still gotten done in hand-coded assembly as often as not.
Basically: with a Z80 or 6502 or 8086, if you had a task "X" to do the "best" way was as often as not the obvious way. With the 386, that was suddenly no longer true and you had to have a bookshelf full of Abrash tomes and whatnot. It was a very different feel.
On the other hand, the 64-bit addressing modes that AMD created with the 64-bit extension are a lot weirder.