This is a good question and I like a non-IANAL response. As far as I can tell police officers assume any resistance to request as proof of guilt. This kind of thinking has happened historically in the past and there's no reason to believe it's not happening now.
I believe searching your car and searching your person constitute two entirely different searches. If your wallet is on the seat next to you, the police could theoretically use that as grounds for a search. If your wallet is in your pocket, and they take your wallet off you as part of a search of your vehicle, you'd probably have a better chance of successfully claiming unreasonable search and seizure.
I'm not a lawyer either though, so take that for what it is worth.
In many places you won't have a chance of doing anything. Just do a quick search in youtube and watch all the people being robbed, beaten, raped, and killed by the police. That's the power they have now.
Now they're asking to see my entire wallet?! I'm assuming we're all well within our rights to refuse to hand over our wallets?