Recurrent neural network language models have solved the problems of data sparseness and dimensionality disaster which exist in traditional N-gram models. RNNLMs have recently demonstrated state-of-the-art performance in speech recognition, machine translation and other tasks. In this paper, we improve the model performance by providing contextual word vectors in association with RNNLMs. This method can reinforce the ability of learning long-distance information using vectors training from Skip-gram model. The experimental results show that the proposed method can improve the perplexity performance significantly on Penn Treebank data. And we further apply the models to speech recognition task on the Wall Street Journal corpora, where we achieve obvious improvements in word-error-rate.