If it works as I understood, in this setup I can see an advantage at an architectural level: in Podman containers images are stored on a per-user basis, while in this setup they would be shared between users, thus using much less disk space (if using the same base images). Besides this, I actually have the same question.
I think OP is referring to the "unprivileged user namespaces" [1] feature of Linux, which caused numerous security incidents in the past. AFAIK, this is mainly because with this feature enabled, unprivilged users can create environments/namespaces which allow them to exploit kernel bugs much more easily. Most of them revolve around broken permission checks (read: root inside container but not outside, yet feature X falsely checks for the permissions _inside_). [2] has a nice list of CVEs caused by unprivileged user namespaces.
Given that rootful docker e.g. is also prone to causing security issues, it's ultimately an attacker model / pick-your-poison situation though.
Ok, but here the OP is doing something a bit different than just rootless Docker, which is to use a "centralised" rootless Docker running as a single, non-privileged user... or am I missing something?