For some use cases, I like the approach of having a fixed number of main, long lived threads/processes at runtime (which matches the number of CPU cores on the machine to minimize context-switching). Where possible, the threads/processes are themseves responsible for picking up the work that must be done. Each thread/process can use hashing/sharding to figure out which portion of the state they are responsible for and only operate on those parts. These kinds of threads should be handling the vast majority of the work. But of course, for this to be possible, the work needs to be parallelizable (servers/sockets make a good use case).
If you have occasional I/O tasks which don't require much CPU (e.g. disk I/O), you can use a separate threadpool for those since the context switching penalty will be minimal. Context switching is mostly a problem when all the threads/processes are using a lot (and equal amount) of CPU. If a very heavy thread/process has to share CPU with a very lightweight thread/process, the context switching penalty is not significant.
- if you have lots of short-lived cpu-intensive tasks then a pool with one or two threads per cpu core (depending on how SMT works for you on your hardware) works well
- if you have lots of long-lived cpu-intensive tasks then just give each task a thread and let the OS schedule them
- if you have lots of io tasks, don't use threads; async io is a massive win
- if you have lots of io tasks on aws then you have to have high core counts even if they all sit idle because of the way credits are divvied up; even with massive brought iops you don't get good io on aws compared to, say, the cheap laptop you do dev on ;)
I am so so so tempted to go into a particular mysql storage engine that my day job often relies upon and move it from one-thread-per-core to async io. Obviously that's a pipe-dream but the wins would be massive (on linux, on aws blah blah)
Having made this list I can see there are so many cravats that I'm not sure generalizations get anyone very far, sadly. Its like the same server software has really different performance characteristics on different cloud providers vs dedicated hardware etc etc.
The exact terminology probably depends on your programming environment as well. I'm using Node.js now and async IO (e.g. for disk access) uses a threadpool behind the scenes but these threads use very little CPU: they're mostly idle in fact; their only two responsibilities are to start the IO operation then send back the data to the parent process when the IO completes so these threads use almost no CPU so they're not a problem but I guess it depends on how heavily they are used. Node.js is not really designed to be a DB engine though so this design works well.
If you have occasional I/O tasks which don't require much CPU (e.g. disk I/O), you can use a separate threadpool for those since the context switching penalty will be minimal. Context switching is mostly a problem when all the threads/processes are using a lot (and equal amount) of CPU. If a very heavy thread/process has to share CPU with a very lightweight thread/process, the context switching penalty is not significant.