"Exactly where I misstepped." He's rather contrite, actually. Embarrassment is a form of harm, after all. "I beg your pardon."

Perhaps it's better to simply change the subject and hope for a more comfortable conversation.

"Are there any other questions I might be able to answer for you? To an extent, you do seem curious about me."