Recurrent neural networks (RNNs) are a widely used deep architecture for\nsequence modeling, generation, and prediction. Despite success in applications\nsuch as machine translation and voice recognition, these stateful models have\nseveral critical shortcomings. Specifically, RNNs generalize poorly over very\nlong sequences, which limits their applicability to many important temporal\nprocessing and time series forecasting problems. For example, RNNs struggle in\nrecognizing complex context free languages (CFLs), never reaching 100% accuracy\non training. One way to address these shortcomings is to couple an RNN with an\nexternal, differentiable memory structure, such as a stack. However,\ndifferentiable memories in prior work have neither been extensively studied on\nCFLs nor tested on sequences longer than those seen in training. The few\nefforts that have studied them have shown that continuous differentiable memory\nstructures yield poor generalization for complex CFLs, making the RNN less\ninterpretable. In this paper, we improve the memory-augmented RNN with\nimportant architectural and state updating mechanisms that ensure that the\nmodel learns to properly balance the use of its latent states with external\nmemory. Our improved RNN models exhibit better generalization performance and\nare able to classify long strings generated by complex hierarchical context\nfree grammars (CFGs). We evaluate our models on CGGs, including the Dyck\nlanguages, as well as on the Penn Treebank language modelling task, and achieve\nstable, robust performance across these benchmarks. Furthermore, we show that\nonly our memory-augmented networks are capable of retaining memory for a longer\nduration up to strings of length 160.\n