Of course it's ego, but it's not undeserved ego, either. For all he gives Charlie hell for being cocky and thinking he's so great, he probably doesn't help matters by also thinking Charlie's pretty great. Maybe that's why he's usually so quick to make fun of Charlie -- he wouldn't want to have him get too carried away with the ego thing.

He can agree that Charlie's wearing too many clothes (to be fair, he usually is, but he can't walk around naked all the time) and he can certainly help tug Charlie's jeans off once Charlie gets them undone. It's strange to him, seeing Charlie dressed so casually, but he doesn't dislike it; it's just another reminder of how different things are here.