My SSD has a three core cpu dedicated to it. Removing the layer of indirection isn't going to make it faster. At best it's going to make the ssd cheaper by a minuscule amount due to a simpler controller.
Its probably fast enough for its purpose. If it isn't, making it faster is way easier than changing existing abstraction layers. In the extreme, it can ever be an asic.