Sounds like you describe the state of Swift at least three years ago. SourceKit just doesn't crash anymore on me anymore, ever. Compile times are quite good for me nowadays, still not as fast as Objective-C but then again you don't get compile time guarantees for free.
The exponential issue was still there in version 4 and I think 5 as well. Didn't say SourceKit crashed, but it was virtually useless because it functioned like a crap version of the language parser.
So what's particularly hard on the compiler are hard coded sets of data that have mixed types. For example a dictionary that reflects a JSON response that also has nested dictionaries.
It helps if you test declare it like:
let json: [String: Any] = [ /** lots of fields here */ ]
String concatenation: could you give me a specific piece of code that fails? It could be that I really never write code that triggers the problem. Using the + operator to concatenate strings has some performance issues though.
I'm not telling you're crazy, I just noticed that things got so much faster in the past two years.
The string concatenation issue was that if you do a ++ b ++ c... When that got to about 6 variables that single line alone took a minute to compile. Go to 10 variables and just give up on ever seeing a result. I used to try it in online Swift REPLs and it was still happening with whatever was most recent a bit over a year ago.