I generally support end-to-end encryption for everything, but I'm not sure that it makes sense in the context of IRC. IRC networks are usually public, so anyone could join your channel and listen in, even with end-to-end encryption. It seems like E2E would make for a lot of complexity and overhead without tangibly increasing the privacy of the users.
Before E2EE was used in IM clients, IRC already had IRC over TLS, and also OTR (which was also used in Gaim/Pidgin).
On IRC, IRC over TLS doesn't have the same threat model as E2EE. With IRC over TLS, the server(s) can read the data plaintext. With proper E2EE (not the marketing version) that's not the case; only clients can read the data. I'm talking about actual data/content here; not metadata.
Hence the "usually" public, I presume. While this doesn't invalidate your point that IRC could use E2E encryption, I personally only use IRC for communication on public channels, where it would be largely pointless, unless you're assuming a really paranoid threat model, in which case public group conversation is probably not a good idea anyway.