Agreed, although a midi controller by its generic nature does have to adapt to very different hardware, so that for example the knobs, switches etc. placement doesn't reflect functionally the hardware it is connected to, which sometimes would help immensely to ease the operators work.
I wish there was a controller made of pluggable micro modules, that is, say 2x2 cm modules receptacles that could host a potentiometer, a pushbutton, a few leds, a sensor pad etc. where no matter where the module is placed, it will link to the backplane processor getting its identification number and function, then the user can move them around to replicate the functional arrangement of an instrument panel to make programming much easier.
It's more or less the same idea I'd like to develop for computer keyboards, where pluggable keys (pots, haptic transducers and sensors etc.) could be moved around still keeping their programmed function. With todays technology it wouldn't be so hard to develop; very likely the most expensive part would be the plugs/receptacles hardware rather than their digital hardware or firmware.
There's a Waterloo start-up that spun out of a Fourth Year Design Project, Palette [0]. Their main sell was integrations into various creative tools (ex. Adobe Creative Suite, Ableton). They recently rebranded as Monogram [1][2] after announcing new hardware.
https://www.kickstarter.com/projects/electro-smith/daisy-an-... is worth a look as well. They have dev platforms in Eurorack, desktop synth, and guitar pedal form factors that could be used for something like this, though they have enough DSP to actually run the audio processing the module, but it seems like it could be used for something like a hybrid setup as well with VCV rack with some software work.
