The presence of a partner can attenuate physiological fear responses, a phenomenon known as social buffering. However, not all individuals are equally sociable. Here we investigated whether social buffering of fear is shaped by sensitivity to social anxiety (social concern) and whether these effects are different in females and males. We collected skin conductance responses (SCRs) and affect ratings of female and male participants when they experienced aversive and neutral sounds alone (alone treatment) or in the presence of an unknown person of the same gender (social treatment). Individual differences in social concern were assessed based on a well-established questionnaire. Our results showed that social concern had a stronger effect on social buffering in females than in males. The lower females scored on social concern, the stronger the SCRs reduction in the social compared to the alone treatment. The effect of social concern on social buffering of fear in females disappeared if participants were paired with a virtual agent instead of a real person. Together, these results showed that social buffering of human fear is shaped by gender and social concern. In females, the presence of virtual agents can buffer fear, irrespective of individual differences in social concern. These findings specify factors that shape the social modulation of human fear, and thus might be relevant for the treatment of anxiety disorders.