Translation tests are widely used for high school term tests and entrance examinations as well as university entrance examinations. Although a considerable number of papers point out the possibility of low reliability for scoring, little is known about the factors raters play in the reliability of scoring (Watanabe, 1994). This study examines how the professional backgrounds of raters affect rating criteria. The results indicated that novice raters tended to over-estimate examinees' comprehension whereas experienced raters were more likely to focus on the correctness of the Japanese sentence. In addition, it turned out that the difficulty of sentences affected the scoring of both experienced and novice raters. After administering a sorting task, the difficulty of the sentences showed that the perception of sentence difficulty did not correspond to the difficulty of examinees' translation. The paper closes by suggesting several pedagogical implications for administering translation tests. Of particular importance is that test developers should consider not only the complexity of sentence structures and vocabulary, but also the examinees' topic familiarity of the sentences to be translated.