Color space conversion is a hard topic in terms of performance. First of all, not all images are stored in sRGB. Most of them have another color profiles (such as P3 or ProPhoto). So, sRGB conversion is not enough, you need the full color management.
Second, you'll see a real profit of color management only on a few images. Most time you'll see the difference only when you see both images at the same time on the same screen.
For now, I came up to the resizing in original non-linear color space and saving the original color profile with the resulting image.
By far most images in the wild are sRGB, and those that aren’t should typically be tagged with their color space.
Resizing in gamma-adjusted space (sometimes) causes nasty artifacts when resizing. If you can afford the CPU use, always convert to an approximately linear space first, then downsize, then convert back. If you get the gamma curve slightly wrong (e.g. gamma = 2.0 vs. 2.2) it’s not too big a deal, the resulting artifacts won’t really be noticeable, so feel free to use a square root routine or something if it has better performance.
But note if you go to a linear space, you need to bump up the precision of your numbers. 8 bits per channel doesn’t cut it anymore, you’ll want 12 or 16 bits per channel (or even better, a floating point number). This might have a big effect on performance.
You don't really need the full color management for resizing, only fixing the gamma curve. I believe Apple's implementation of P3 uses the common 2.2 gamma, which is also approximately what sRGB uses. Unfortunately ProPhoto uses 1.8.
Second, you'll see a real profit of color management only on a few images. Most time you'll see the difference only when you see both images at the same time on the same screen.
For now, I came up to the resizing in original non-linear color space and saving the original color profile with the resulting image.