It's a best effort though, you're relying on the package maintainer to have tracked all the installed files and clean up anything they might have modified, which hopefully you or something else hasn't modified after the package was installed.
The packages authored by distributions tend to be pretty good, but I've seen RPMs with nothing more than tar zxf in them and no tracking of the files installed.
The major package managers expect the package to manage the artifacts installed, this works if everyone plays nice and does their job...
You're right that this requires well-maintained packages, and this is one of the strengths of Debian: its policies that require well-maintained packages. If you install a package that's part of Debian, you can count on it being well-maintained, as opposed to one put out by a random web site or developer, which may install but may not integrate well or clean up after itself.
Package formats each have their own expectations. RPM likes being pointed to a source tarball, given the steps to build the project, and being given a list of the just-built files to include in the package. Those are the artifacts that RPM will manage (and they can be marked things like "doc", "config", or given specific attributes). Deb is broadly similar, but I feel like the framework is a little less controlled than the RPM one.
The manager basically just defines a framework. Packages can abuse that framework in various ways (I haven't even mentioned install triggers, to execute code when other named packages are added or removed). There are a lot of ways to get it wrong, but when the dev builds a good package, it's almost magic how nicely it works.
The packages authored by distributions tend to be pretty good, but I've seen RPMs with nothing more than tar zxf in them and no tracking of the files installed.
The major package managers expect the package to manage the artifacts installed, this works if everyone plays nice and does their job...