Depends on your platform, there are some platforms where the compositor/hardware demand ARGB and some where they demand RGBA. As a result if your backbuffer/etc are not in that format you eat some overhead.
This isn't a big justification for it in a texture format, though. A channel swap isn't that expensive to do at load time and the driver can often do it for you during the upload depending on the API you're using. If you really want to load blazing fast your textures should be pre-compressed in hardware formats that you can mmap in, not stored in QOI or PNG.
