Temporal models based on recurrent neural networks have proven to be quite\npowerful in a wide variety of applications. However, training these models\noften relies on back-propagation through time, which entails unfolding the\nnetwork over many time steps, making the process of conducting credit\nassignment considerably more challenging. Furthermore, the nature of\nback-propagation itself does not permit the use of non-differentiable\nactivation functions and is inherently sequential, making parallelization of\nthe underlying training process difficult. Here, we propose the Parallel\nTemporal Neural Coding Network (P-TNCN), a biologically inspired model trained\nby the learning algorithm we call Local Representation Alignment. It aims to\nresolve the difficulties and problems that plague recurrent networks trained by\nback-propagation through time. The architecture requires neither unrolling in\ntime nor the derivatives of its internal activation functions. We compare our\nmodel and learning procedure to other back-propagation through time\nalternatives (which also tend to be computationally expensive), including\nreal-time recurrent learning, echo state networks, and unbiased online\nrecurrent optimization. We show that it outperforms these on sequence modeling\nbenchmarks such as Bouncing MNIST, a new benchmark we denote as Bouncing\nNotMNIST, and Penn Treebank. Notably, our approach can in some instances\noutperform full back-propagation through time as well as variants such as\nsparse attentive back-tracking. Significantly, the hidden unit correction phase\nof P-TNCN allows it to adapt to new datasets even if its synaptic weights are\nheld fixed (zero-shot adaptation) and facilitates retention of prior generative\nknowledge when faced with a task sequence. We present results that show the\nP-TNCN's ability to conduct zero-shot adaptation and online continual sequence\nmodeling.\n