No, I won't leave Facebook entirely over this, but I did just uninstall their application on my Android device. I simply don't trust them to have access to my contact list.
Frankly, I didn't trust them much before: I'm sure they were mining it, which is not something I would explicitly consent to. However, now I no longer trust them to not break it, so I've removed their access to it entirely.
I see they actually created a way to remove the FB app from an Android device. Back when they hoovered up phone contacts into their database, I tried to break the connection, but all I could do at the time was delete my entire facebook account. Been pretty happy not wasting my time there, as I see it devolving into pseudonymous games, meme one-upmanship, and picture albums kids will mostly later regret.
If you installed Facebook yourself, you can always remove it. You can't remove it if your carrier or manufacturer put it there (at least not withput rooting). AFAIK Android 4.0 allows you to deactivate these apps, effectively removing them witgout deleting the files.
That assumes you CAN uninstall it. Many devices come with Facebook preinstalled and not removable (because Facebook worked a deal with the device manufacturer or the cell company).
Frankly, I didn't trust them much before: I'm sure they were mining it, which is not something I would explicitly consent to. However, now I no longer trust them to not break it, so I've removed their access to it entirely.