I'm not sure about other Americans, but I'll sometimes say "swim[ming] trunks" if I'm specifically referring to male swimming attire. Otherwise, yeah, it's often "swim[ming] suit" or "bathing suit" here.
Budgie smugglers does indeed only apply to the brief style, but those who like banter and wordplay will try to work it into any conversation involving male swimwear. E.g. "Fetch your budgie smugglers fellas, we're going to the beach", or "sorry ladies, my budgie smuggling days are over. It's boardies for me."
Since male swimming attire comes in different shapes (ahem designs), I hear people use «swimmers» more and more.
Save for one singular design which, it would seem, distinguishes itself with such remarkable uniqueness as to warrant the bestowal of its very own appellation: «budgie smugglers» (AusE).
