Recurrent Neural Networks (RNNs) play a major role in the field of sequential\nlearning, and have outperformed traditional algorithms on many benchmarks.\nTraining deep RNNs still remains a challenge, and most of the state-of-the-art\nmodels are structured with a transition depth of 2-4 layers. Recurrent Highway\nNetworks (RHNs) were introduced in order to tackle this issue. These have\nachieved state-of-the-art performance on a few benchmarks using a depth of 10\nlayers. However, the performance of this architecture suffers from a\nbottleneck, and ceases to improve when an attempt is made to add more layers.\nIn this work, we analyze the causes for this, and postulate that the main\nsource is the way that the information flows through time. We introduce a novel\nand simple variation for the RHN cell, called Highway State Gating (HSG), which\nallows adding more layers, while continuing to improve performance. By using a\ngating mechanism for the state, we allow the net to "choose" whether to pass\ninformation directly through time, or to gate it. This mechanism also allows\nthe gradient to back-propagate directly through time and, therefore, results in\na slightly faster convergence. We use the Penn Treebank (PTB) dataset as a\nplatform for empirical proof of concept. Empirical results show that the\nimprovement due to Highway State Gating is for all depths, and as the depth\nincreases, the improvement also increases.\n