Yeah, type inference shouldn't really take a huge amount of time like that - for example Rust's type inference is a pretty small part of the overall compile time (although it can take longer if you do some crazy stuff with traits).
In Swift's case it seems like they've tried to add very flexible type inference (leaving off lots of stuff) to a very flexible type system (with inheritance, overloading, protocols, etc.). This is something that academics have been aware for decades causes problems with type checking performance. Similar problems have caused the designers of Scala 3 to go back to the drawing board to redesign the type system from the ground up, with a clearer formal semantics.
I do know that the Swift team are working really hard to make on demand editing/compiling better, but I don't know how far that work has come. Hopefully that improves things over time!
In Swift's case it seems like they've tried to add very flexible type inference (leaving off lots of stuff) to a very flexible type system (with inheritance, overloading, protocols, etc.). This is something that academics have been aware for decades causes problems with type checking performance. Similar problems have caused the designers of Scala 3 to go back to the drawing board to redesign the type system from the ground up, with a clearer formal semantics.
I do know that the Swift team are working really hard to make on demand editing/compiling better, but I don't know how far that work has come. Hopefully that improves things over time!