I fully agree. But: I believe in many cases this is because, with both Android Auto and Apple CarPlay as I have used both for extended periods, they are simply buggy as hell.

Some car vendors have better implementations than others, but I frequently see things happen like the connection suddenly cuts or audio playback becomes choppy or sometimes the dash crashes entirely.

I suspect if it was less buggy it would be a lot less distracting, personally.