I doubt he is trolling. I think you annoyed him once, that colored his view of you, and now he evaluates your actions in that context and acts accordingly. Somewhat irrational, but that's how people are.