I tend to agree with what you’re saying. My impression is that the main concern is for current technology used by astronomers, satellites, navigation and so on, which require leap seconds to be taken into account and have a relation to the length of day, and which would stop working correctly if the meaning of broken-down time would suddenly change. It maybe wouldn’t be a major problem if all such systems could be redesigned from scratch.
