yeah the way to use slack (for distributed teams) is the same as the way you use the water cooler (for office-based teams). you only hear what you're there to hear. if someone put an audio recorder at your office cooler you'd never listen to the recording.
however, i think that's their entire point. it's actually not very useful for actual topical discussions
It depends on the work place, but for the agency I used to work at you would be working on a particular client's project on any given day and therefore you only needed to tune into one channel. But more than that, any decisions or discussions particular to a task happened in the ticket for that task, not in slack. Slack was used for short periods of quick deliberation and problem solving environments or configuration, or general state of the project type conversations. In other words if it didn't fit in a ticket it went into slack.
That worked really well for isolating slack communication to silos of attention and it solved the problem of information important to a task living in slack.
Ultimately it's not the tools fault for any of this though. It's just a bunch of chat channels. It's up to you and your company to form good policies and practices.
Maybe that would be a compelling idea: have a channel-level chat for those ephemeral conversations, where the messages disappear after a certain number of minutes/hours/days/messages (maybe configured per-channel), and then have named topics within the channels for more persistent/ongoing conversations.
Maybe Zulip already does that. Haven't tried it yet.
however, i think that's their entire point. it's actually not very useful for actual topical discussions