Addressing modes on the 68020 are kind of crazy. In addition to some relatively straightforward improvements (scaling for the indexed mode, options for larger displacements on both the displacement mode and indexed mode) they also added something called "memory indirect" modes. These allowed you to dereference a pointer in memory in a single operation. In these modes you have a base register, a base displacement, an index register (with scale) and an outer displacement. The index register could be applied either to the base value or to the fetched "outer" value.
Yeah, that's exactly what I was thinking of and why I had to reread the original post twice. The 68020+ was perhaps crazier than the VAX. The last two variants you mentioned were called preindexed and postindexed mode.
I wrote a few hundred thousands of lines of 68000 code, but much less for 68020+. Those were the days!
For clarity: I mentioned the 386 addressing modes because fundamentally the ModRM encoding was designed to address the same code generation problem: efficiently encoding one instruction to compute base-plus-offset (-plus-immediate too) in addressing memory. This avoids having to compute an address first for what is one of the most common operations in application code. As it happened, Intel's trick was the better idea. Motorola's original register design was fine but not as good, the '020 madness didn't survive contact with the RISC pipeline.
With the exception of index scaling (which as already mentioned, was added in the 68020) ModR/M and SIB is a strict subset of the 68000 addressing modes. I don't see what this has to do with the address register/data register split though. The 386 only had 8 GPRs and only 7 of those could be used as a base or index register. The reason for the address/data split is to allow 16 registers without needing 4-bit register fields.
Apart from the overly complex memory indirect modes, I'm having a hard time seeing how the 386 ModR/M and SIB setup is superior to what was in the 68020. Twice as many registers (though with usage restrictions), PC-relative addressing and a cleaner encoding. Those first two things have been fixed in x86-64 (and with generally fewer usage restrictions), but at the cost of making the encoding even worse.
