I trust the package managers to build software in a way that I can be sure matches with the published source package. That's the important difference. Projects bundle blobs into their binaries that aren't in the source code all the time (cf. chromium as discussed, but firefox did (does?) this too).
Of course you could say in a large codebase like chromium, bad things can be hidden. But at least there's a better chance such things are found, and it's more risky to put it there in plain sight.
In a perfect world, all software would have reproducible builds and there would be no issue with trusting that binaries contained no hidden functionality not in the source, as there'd be third parties to rebuild and compare the result against the published binaries. We're a ways away from that.
Of course you could say in a large codebase like chromium, bad things can be hidden. But at least there's a better chance such things are found, and it's more risky to put it there in plain sight.
In a perfect world, all software would have reproducible builds and there would be no issue with trusting that binaries contained no hidden functionality not in the source, as there'd be third parties to rebuild and compare the result against the published binaries. We're a ways away from that.