I think this is a function of hours of practice. I remember the point when I started hearing the silent "h" in Spanish and also when I stopped understanding what native speakers were saying after 3 months because I'd progressed beyond the level where they felt they had to slow their speech down for me. Lots of plateaus in the learning curve.