As I've said before on other threads, hearing loss is cumulative. In other words, the damage builds up over time and can eventually be quite severe. Loud sounds above 90-100 db cause hearing loss, period. The extent of the loss and the frequencies lost are different from person to person. Loud music, industrial noise, traffic noise and gunfire are some sources of noise above the 90-100 levels and will cause damage. Gunfire is well above 165 db and should be avoided.
I wear protection even when I'm hunting at least on my strong side but I don't hunt much large game anymore so sight hunting is fine for me.
I suppose the sound wave could be more severe in a confined space such as a car, you would have to get a meter to determine the increase in levels though. Shooting in self defense is a situation that can not be avoided and worrying about your hearing is something you won't be thinking about!