You're right that a tool which runs under the same user could read your config file and thus could access to your secrets.
But there is one main difference: that tool would need to do so explicitly, with the intent of reading (and possibly exposing) your secrets. For me, that's a huge difference from having the secrets being implicitly available to the process through the processes environment.
It doesn't need to do so explicitly: it just needs to have a bug that can encourage it to do so. Most environment leakage also needs to be triggered too.
But there is one main difference: that tool would need to do so explicitly, with the intent of reading (and possibly exposing) your secrets. For me, that's a huge difference from having the secrets being implicitly available to the process through the processes environment.