"I've wanted to the whole time I've been talking to you."

He's still not yanking his wrist away, though. Maybe he's trying to prove a point, too, that he can go toe to toe with Charlie and face him down, that he's not intimidated by him. Charlie's temper may irritate him, may goad him into reacting in ways that he'd never react to anyone else, but it doesn't scare him.