Something I didn't mention in my original comment, but have mentioned in another reply somewhere, is that I have the websites running behind Cloudflare, and I allow Cloudflare's ASN into port 443 but block everything else.
Essentially outsourcing the security of port 443 to Cloudflare.
My use-case is "hobby / enthusiast", so I believe I'm losing nothing and the "world at large" is losing nothing from this setup. Having said that, all policies on this kind of thing need to be strongly thought about in terms of their applicability to the use-case.
Were I running a small or even medium business, I'd probably do it exactly the same with maybe a bit more of an eye on what's being blocked and the ownership of the IP addresses, and I'd have some stats to point to on the range of sources of legitimate traffic. It'd have to be a pretty big, international business for it to cause much of an effect (although I'm talking well out of school here because I don't have anything at stake).
Flipside, though, I have my outgoing traffic routed through a couple of different exits, and I've had to make specific rules for some websites that block traffic from VPNs and VPSs, which is annoying, so I'm not completely dismissing your point.
Lastly, however, at all scales I'd still block the Internet Scanners for reasons I've given elsewhere. Blocking them massively cut down on the uninvited activity - again, it's not about making clean logs, but it really helped clear a lot of the noise.
This means that you are locking out anybody using a paid VPN service, if any other customer of that same VPN service does any kind of scan.