This is why I never give permission to apps to access my contacts. I wish Apple would have better protections than just a simple permission ask, when once granted, apps can do whatever they want with all of that data.
It's what I hate about whatsapp, I have to use it so I have it installed on my phone but I have to jump through so many hoops to properly isolate it from my contacts.
Unfortunately, I can't not have it, it's too entrenched because most people do not care about privacy.