A link to info about SELinux? It's a rather broad topic, you'll probably be better served to google for what ever aspect of it you're interested in but it's definitely in Redhat (even turned on by default [1]) and it's on Debian (not sure if it's on by default there).
About how to use SELinux to ensure that a process with certain PID is running the intended executable and that it hasn't been tampered by the user (i.e. a rogue app running with user perms).
Well, you'd need to read up on how SELinux works to understand that.
Briefly, SELinux adds some new permissions to files, new contexts to users and so on, so what happens is you specify what a given context is allowed to do, to a very granular level. For example, if you take postfix, it's split up into lots of programs, so you could say that the "init" context is allowed to run the postfix startup script but none of the rest of it. The postfix startup script will convert into some "postfix startup" context which is allowed to start the various programs but nothing else. So there will be a a postfix process that allows incoming IP addresses and can send on a specific FIFO and nothing else. There is another program which can read from the FIFO, change certain files and nothing else. So, imagine if you break into the program that talks with the internet: all you can do at that point is talk with the internet and talk with the FIFO. You can't read any files, write anything, execute anything, etc.
So that's kind of the idea of it. The above assumes an extremely tight SELinux configuration and no bugs in SELinux that would allow you to break out of your context. I've never seen a configuration as tight as described above but there did use to be a guy who had a Debian linux machine running on the internet where you could log in as root and try to break stuff. As far as I know, no one ever hacked that machine, starting out as root in a bash shell.
I fail to see how it can help in this case. The most fine-grained access you can achieve with SELinux are objects such as files or ports. What you need here is the ability to check if the other process has the permission to invoke certain operations from the current server through the Wayland protocol. It would be like arbitrary capabilities, but not linked to the operating system but to specific applications.
For example, in order for the Color Picker Tool to work, The Gimp should be marked with a "color picked allowed" capability, so when it asks the Wayland server for the color of pixels outside the surfaces it already owns, the server can check it and send the requested info. But a rogue program/process trying to scrap the screen content pixel by pixel shouldn't be able to do that. The inability to safely map processes to executables in Unix (and the possibility of manipulating their running code via exec(), library injection, ...) make it a very hard problem to solve without a paradigm shift that SELinux doesn't provide (as far as I can tell).
Oh I see. Yes, I don't think SELinux could do exactly the scenario you decide because the things you're mentioning don't exist at the OS level. If you can figure out a way to make them something SELinux can get a context onto then it would be possible.
