"I was never held accountable for everything I did. Yes, you and Dean were angry with me and eventually forgave me, but that was for the crimes I committed against you specifically."

It's difficult to talk about this, even now. But Castiel owes it to Sam. Maybe that, along with what happened last night, can be part of his punishment.

"But I haven't been punished for what I did to Heaven, to Earth. For the thousands whose lives I took."

And yet he's unaware that no one in the world can punish him as hard as he punishes himself. That, maybe, is something Sam can relate to better than anyone else.