That sounds like a roundabout way of saying "filter bubble". Having to defend your opinions once in a while isn't all bad. If you have to surround yourself with people you find intelligent you might just be biasing your social circle towards people who agree with you a priori.
That is a definent concern, but I think it is possible to separate "intelligent" from "agrees with me".
For me, the biggest requirement to having an intelligent conversation is the ability to distinguish questions of fact, from questions of values. By far the most intelligent discussion group I have participated in, had me in the minority opinion in most discussions.
It's good to be able to distinguish different things. In my experience, however, reasonable people can disagree on questions of fact, on questions of value, or on both questions. It is an extremist position to require unanimity on either "facts" or "values".
This is a good point but I was not suggesting surrounding yourself with people who agree with you. I meant people who disagree with you without getting emotionally invested or spouting logical fallacies.