The effects of lexical difficulty and talker variability on word recognition were examined in four groups of listeners: native English/normal hearing; native English/hearing impaired; non-native English/normal hearing; and non-native English/hearing impaired (hearing level matched to the native hearing impaired). Lexical difficulty was measured by the difference in performance to 75 lexically ‘‘easy’’ and ‘‘hard’’ words based on word frequency and Neighborhood Activation Theory [Luce and Pisoni (1998)]. The effect of talker variability was measured by the difference in performance between single and multiple talker (nine talkers) conditions. The familiarity of the 150 words was rated on a seven-point scale. An up–down adaptive procedure was used to determine the sound pressure level for 50% performance. Non-native listeners in both normal and hearing-impaired groups required a greater intensity for equal intelligibility than for the comparative native normal and hearing-impaired listeners. Results, however, showed significant effects of lexical difficulty and talker variability in all four groups. Structural equation modeling demonstrated that an auditory factor estimated by pure tone average, etc., accounts for four times more variance to performance than does a linguistic fluency factor measured by word familiarity ratings and native versus non-native status, however, the linguistic fluency factor is also essential to the model fit.