At the end of the day people don't care about security that much. Wayland is still getting there. Its getting better, particularly in the last year, but its not a huge priority for many people so it has taken upwards of ten years to become usable.
I think many people are also not aware of how bad X11 security is. The other day I mentioned that X11 applications can read all keystrokes to someone who does system administration/security at an ISP. To my big surprise, he was not aware of this. (Didn't everyone play with xev at some point?)
On Linux, I only use Wayland because of X11's broken security model. Fortunately, it works very well these days with the amdgpu driver.
However the way Wayland (or its broader ecosystem) "fixes" security is just stupid. Instead of using a fine grained whitelisting based permission system to grant different clients all the permissions they need to do their work (taking screenshots, reading the color of a point, listening to a certain key combination, apply some filtering to the display, get the size/position of a different client, ...) it treats every client as untrustworthy with no standard way to change that.
And let's not forget that just using Wayland doesn't grant any improved security compared to X11. There are numerous other ways to get input data, display contents, etc., so you need further sandboxing anyway, at which point you might just as well sandbox X11 clients you don't trust to get the same security. The benefit here is that you can decide if you trust your color picker and let it do its work by not sandboxing it, while Wayland just decides for you that no color picker can be trusted.
> However the way Wayland (or its broader ecosystem) "fixes" security is just stupid. Instead of using a fine grained whitelisting based permission system to grant different clients all the permissions they need to do their work
Because you can't do that in Unix. You can't ensure that the process with PID 1234 is really executing /usr/bin/foo.
Assuming other apps are sandboxed from /proc/1234, can't the shell just exec "/usr/bin/foo" and pass some file descriptors or secrets in environment variables?
How the wayland server and the shell are going to share (or agree to) that file descriptors or secrets in the first place? Or how can you ensure that the shell is not a rogue one?
> How the wayland server and the shell are going to share (or agree to) that file descriptors or secrets in the first place?
I read that GNOME just implements both in the same program (and calls it "shell").
> Or how can you ensure that the shell is not a rogue one?
I'm confused. Do you mean this kind of attack would become somewhat more probable or more dangerous if the original shell had some mechanism of allowing certain apps to pick colors?
The reason that Wayland (or X11) protocol uses Unix sockets for communications is because there is not a hierarchical relationship between a client process and the server process (they can be even from different user "sessions"). Also, a client can be launched from anywhere, including a "shell" such as a /bin/bash interactive session.
A link to info about SELinux? It's a rather broad topic, you'll probably be better served to google for what ever aspect of it you're interested in but it's definitely in Redhat (even turned on by default [1]) and it's on Debian (not sure if it's on by default there).
About how to use SELinux to ensure that a process with certain PID is running the intended executable and that it hasn't been tampered by the user (i.e. a rogue app running with user perms).
Well, you'd need to read up on how SELinux works to understand that.
Briefly, SELinux adds some new permissions to files, new contexts to users and so on, so what happens is you specify what a given context is allowed to do, to a very granular level. For example, if you take postfix, it's split up into lots of programs, so you could say that the "init" context is allowed to run the postfix startup script but none of the rest of it. The postfix startup script will convert into some "postfix startup" context which is allowed to start the various programs but nothing else. So there will be a a postfix process that allows incoming IP addresses and can send on a specific FIFO and nothing else. There is another program which can read from the FIFO, change certain files and nothing else. So, imagine if you break into the program that talks with the internet: all you can do at that point is talk with the internet and talk with the FIFO. You can't read any files, write anything, execute anything, etc.
So that's kind of the idea of it. The above assumes an extremely tight SELinux configuration and no bugs in SELinux that would allow you to break out of your context. I've never seen a configuration as tight as described above but there did use to be a guy who had a Debian linux machine running on the internet where you could log in as root and try to break stuff. As far as I know, no one ever hacked that machine, starting out as root in a bash shell.
I fail to see how it can help in this case. The most fine-grained access you can achieve with SELinux are objects such as files or ports. What you need here is the ability to check if the other process has the permission to invoke certain operations from the current server through the Wayland protocol. It would be like arbitrary capabilities, but not linked to the operating system but to specific applications.
For example, in order for the Color Picker Tool to work, The Gimp should be marked with a "color picked allowed" capability, so when it asks the Wayland server for the color of pixels outside the surfaces it already owns, the server can check it and send the requested info. But a rogue program/process trying to scrap the screen content pixel by pixel shouldn't be able to do that. The inability to safely map processes to executables in Unix (and the possibility of manipulating their running code via exec(), library injection, ...) make it a very hard problem to solve without a paradigm shift that SELinux doesn't provide (as far as I can tell).
Oh I see. Yes, I don't think SELinux could do exactly the scenario you decide because the things you're mentioning don't exist at the OS level. If you can figure out a way to make them something SELinux can get a context onto then it would be possible.
I think many people are also not aware of how bad X11 security is. The other day I mentioned that X11 applications can read all keystrokes to someone who does system administration/security at an ISP. To my big surprise, he was not aware of this. (Didn't everyone play with xev at some point?)
On Linux, I only use Wayland because of X11's broken security model. Fortunately, it works very well these days with the amdgpu driver.