In the case of alcohol the spinning is mostly due to differences in the density of the liquid inside the cilia of your inner ear cells when alcohol gets in and out of them, specially in the second case (some hours after drinking). It's really an alcohol-specific effect. Some other toxins might induce vertigo through other mechanisms though.
