The assertion, or maybe idea, that a new architecture may be the thing is kind of about building AGI rather than chatbots.

Like humans think about things and learn which may require some differences from feed the internet in to pre-train your transformer.