There were plenty of people who had questioned Sam's heart. It was part of the reason he still questioned it himself. It was just that nobody who mattered to him ever questioned it any more.

"Well, it's a bit heavy for using most of the time. But if the bunker were attacked, I might grab it." Which was always a possibility. After all, they had a Knight of Hell on their bad side.