Many NLP applications, such as biomedical data and technical support, have\n10-100 million tokens of in-domain data and limited computational resources for\nlearning from it. How should we train a language model in this scenario? Most\nlanguage modeling research considers either a small dataset with a closed\nvocabulary (like the standard 1 million token Penn Treebank), or the whole web\nwith byte-pair encoding. We show that for our target setting in English,\ninitialising and freezing input embeddings using in-domain data can improve\nlanguage model performance by providing a useful representation of rare words,\nand this pattern holds across several different domains. In the process, we\nshow that the standard convention of tying input and output embeddings does not\nimprove perplexity when initializing with embeddings trained on in-domain data.\n