Either offensiveness is completely subjective in which case it is sole, entirely and complete the concern of the individual claiming to be offended.

Or there is some standard for offensive remarks, in which case you absolutely do get to tell people that they aren't offended by that.

Note that if offense is subjective you don't get to complain about others making you offended -- that is by definition your own feelings.