Test score variability associated with characteristics of raters is considered as a construct-irrelevant variance in test scores. Several studies have tried to identify the effect of rater characteristics on test score variances in performance-based oral communication tests. However, little attention has been paid to the effect of raters’ familiarity with speech topics in an oral communication test. This study investigates how raters’ prior knowledge of test takers’ presentation topics affects rater severity in a teaching simulation test for international teaching assistants. Rating data of 162 test takers by 15 raters at a U.S. university were analysed by using a many-facet Rasch measurement (MFRM) analysis. A three-facet (i.e., examinee, rater, and topic familiarity) rating scale model with MFRM framework was fitted to estimate the magnitude of topic familiarity on rating severity. It was found that raters were harsher when they were less familiar with test takers’ speech topic. The difficulty difference between high (or mid) and low topic familiarity was 0.56 (or 0.53) in logits. There was found to be no interaction between raters’ topic familiarity and their background (NS vs NNS). The results of this study suggest that solid rater training is needed to reduce construct-irrelevant variances in test scores.