Machine learning relies heavily on language modeling to understand detailed information in modern applications of natural language processing. In addition, Deep learning methodologies have gained traction in various fields. Recurrent neural networks have emerged as powerful tools in this context, excelling at sequence modeling. Notably, Long-Short-Term Memory (LSTM) layers have become fundamental to language modeling. Additionally, Temporal Convolutional Networks (TCNs) are a recent and promising addition to Deep Learning. Meanwhile, meta-heuristic optimization has been adopted in several research areas due to its effectiveness. This study proposes a TCN-based language model, optimized by the Arithmetic Optimization Algorithm (AOA). AOA is used in this study to optimize parameters for LSTM and TCN models. The proposed language model is compared with the Exponential Linear Unit LSTM and the Binary Input Gate Recurrent Unit, two important models in modern language processing. The performance of these models is examined and compared using the Penn Treebank (PTB) dataset. The obtained results highlight the ability of AOA to accelerate the convergence of parameters during training and highlight the better performance of the proposed model. By improving Sparse Categorical Cross Entropy loss and perplexity, the proposed model outperforms other models in the PTB language modeling task.