PCIe costs you less than a microsecond of latency. A good SSD has 60 microseconds of latency. You're not going to notice any difference from moving the controller.