I'm not saying you should say yes to be agreeable. Just be polite but firm. I would say something along the lines of "I don't want to give out my phone number sorry." or even just shaking your head back and forth works.
"Do you have a phone number" "No" comes off as you being irritated and will make the staff member feel like a dick for just doing their job.
I'm not saying you must always be polite. If somebody is being a pushy asshole push back. However if you can take a few seconds to make a minimum wage drone not feel as bad about their job its well worth it.
