It usually moves it for good reason, and most bad reasons were eventually weeded out of the scheduler.
For example if task previously running on core is waiting for something, the choice is to allow something else to run there (and hope it finished before that other thread comes back from waiting), or not use it at all and waste CPU capacity.
Obviously scheduler will choose to use that now-idle core, but that might cause the previous process (which scheduler have no idea when it will come back) to be resheduled somewhere else, usually on its sibling (as in closest in memory hierarchy) core.
Or power saving. If say core was idle (thread was waiting on something), it might actually be quicker to run it on other core (especially one close in cache hierarchy) than to wake up the core and then ramp up the clock speed of it.
How about running your process in a cgroup attached to a cpuset with the cores/NUMA nodes you want? How about cgroups v2 which has the threaded mode so a process can put it's threads into different subgroups?
