The comment had, in fact, been meant as a dry joke, but Castiel quickly remembers his place with the way silence falls for a moment. He's been sheltered, and doesn't always know how to talk to actual people - aside from his uncle's condescension towards those of 'lower birth'. Castiel learned etiquette, not social interaction. It's not surprising to him that his humor doesn't ring with these people.

"I'm not a glass figurine", he mutters under his breath, though he refrains from pointing that out. It's hardly surprising that these people would see him like that, and Castiel is beginning to realize that having been the victim in every way so far, he'll have to work on proving to them that he's not a fragile, naive flower maiden.

He doesn't break easy... although tonight under Sam, he wonders if he might.