My understanding is that there are no lead fumes. It becomes a gas at like 3200 degrees. Your soldering iron won’t make it into a gas and if you are inhaling fumes that hot you have bigger problems. My main issue with it is that I don’t want any residue on my work surfaces where my kids might play with it, or on any electronics they might touch.
Fumbling around with a soldering iron and some leaded solder a little bit, few times a decade, won't give you cancer or any poisoning. Safety considerations are aimed at people doing lots of soldering, because at this point the amount of flux fumes they might breathe in (or risk of ingesting lead) becomes a possible risk.
(And to be honest, I believe most of this safety advice is aimed not at DIY people, and not even at people who occasionally have to solder something at work, but to have a stick that will prevent employers in soldering shops from creating very unhealthy conditions for their workers.)
