There's no need to remove addresses from places that already have them, and a system like [Plus codes](https://plus.codes/) could identify physical locations without an address.