The understanding of nonverbal language is essential for communication quality. Emotion recognition from the nonverbal language guarantees adaptation in the social environment and even a professional success, it gives essential information about one’s emotions, social status, attitude, etc. Unfortunately, voice as a source of nonverbal language has been left behind by scientists who mostly concentrate on facial expression. The aim of the current research is to investigate how emotion recognition from voice depends on the receiver’s gender as well as on the gender and age of a person expressing them. The following hypotheses were stated: a) anger and sadness are recognized most precisely while joy least precisely; b) female students recognize emotion from voice better than male students do; c) female and male participants recognize emotion from voice better when it is expressed by the actor of the same gender; d) students can recognize emotions from voice better when they are expressed by an actor of the same age; e) emotions are recognized better when they are considered to match the actor’s gender. 223 students participated in the research. The average age was 21 (SD = 1.8), of them 183 women and 40 men. The participants were asked to listen to a voice recording prepared for this study and to recognize the displayed emotion (joy, boredom, anger, sadness, interest, and fear). Each emotion was portrayed by four professional actors: two men (72 and 23 years old) and 2 women (56 and 24 years old). They were all using the same emotionally neutral sentence: ”I am going out of the room now, but I will be back later”. The method was created based on the method of Byron (2007) used in her research. The results of the current research showed that all emotions were recognized at the accuracy of 33–92%. Anger (99%) and joy (82%) were recognized most accurately, while sadness was recognized least accurately (58%). Women tended to recognize all the emotions more accurately than men, even though only fear and interest were recognized statistically significantly better by women. Investigating the recognition of the positive and the negative emotion groups, women recognized them both significantly better than men did. Moreover, women recognized the emotion portrayed by females better, while men recognized emotions expressed by females and males with the same accuracy. Emotions portrayed by an older actor were recognized better than those portrayed by young actors. Participants of more various occupations and age should be investigated, as well as the emotions portrayed by more different actors and actresses should be displayed in future studies to improve the research of emotion recognition from voice. This study was a great contribution to creating the Lithuanian instrument for evaluating the emotion recognition from voice.