I quoted from Wiktionary, feel free to update the definition there (with sources) if you disagree. Note that the "Synonym: toxic" appears underneath the second (figurative) definition, and in my opinion is perfectly correct; in figurative use poisonous and toxic are used in the same way.
Poisons are something you do to yourself (usually by ingestion), venoms are injected by the other organism (such as a snake bite).