"I didn't say you were an idiot, I said you were impulsive. Maybe if you learned to listen a little better, you'd've got that."

He's just going for the jugular with his insults today, because he can't go for the jugular literally. Or rather, apparently he could, and Charlie would even encourage it, but he can't understand why Charlie would willingly allow him to do something like that.

He's going to ignore the pillow throwing. That's the least of his worries.