I have seen Wiggtenstein's language games invoked to explain this "your red isn't my red" possibility, but I've never really been able to follow the reasoning.

Perhaps some philosophically inclined HNer who passes by here can let me know if this is a legit application of his ideas?