In this paper, we introduce the novel concept of densely connected layers\ninto recurrent neural networks. We evaluate our proposed architecture on the\nPenn Treebank language modeling task. We show that we can obtain similar\nperplexity scores with six times fewer parameters compared to a standard\nstacked 2-layer LSTM model trained with dropout (Zaremba et al. 2014). In\ncontrast with the current usage of skip connections, we show that densely\nconnecting only a few stacked layers with skip connections already yields\nsignificant perplexity reductions.\n