Oh, I missed your comment about being able to tune some wristwatches quartz! I wasn't aware that was a thing.
Still, wouldn't the temperature of a watch while being worn vary as least as much as when sitting in a drawer (unless you live in a region blessed with t-shirt weather year around)?
One of my favorite wristwatches I used to wear as a teenager had a thermometer, but I don't remember how exactly that varied over the year, just that it always showed neither quite my body temperature, nor quite the ambient one :)
The crystals used in watches are usually cut and selected so that a local minimum or local maximum of the tempco is near the temperature of your wrist.
Thus, the tempco is near zero, so human-to-human differences don’t matter much.
One thing to notice is that quartz watches almost always have a metal backplate touching your wrist so that the crystal can have good thermal contact. Presumably, the thermometer in your watch was decoupled from that plate.
Or are you saying that what makes quartz crystals drift is the change in temperature?