Can I see GPT as some sort of lossy compression?

You feed it loads and loads of data, where the neural net basically compresses it all into the structure and weights.

Then you give it some (un)compressed part, and it gives you the other.