Welcome to the Real World

I just told a coworker that he is the worst person I know. I don't know if that was such a good thing to say. He took it as a compliment, so of course that got me even more fired up. I guess in the real world the way people joke is by making condescending comments, which is not the same as friendly banter. He made some comments that stabbed at my ego... I probably should have just said that I did not appreciate the way he jokes and not straight up say he is the worst person I know. Sometimes I enjoy working with him, but for the most part, I do not.