Machine learning training methods depend plentifully and intricately on\nhyperparameters, motivating automated strategies for their optimisation. Many\nexisting algorithms restart training for each new hyperparameter choice, at\nconsiderable computational cost. Some hypergradient-based one-pass methods\nexist, but these either cannot be applied to arbitrary optimiser\nhyperparameters (such as learning rates and momenta) or take several times\nlonger to train than their base models. We extend these existing methods to\ndevelop an approximate hypergradient-based hyperparameter optimiser which is\napplicable to any continuous hyperparameter appearing in a differentiable model\nweight update, yet requires only one training episode, with no restarts. We\nalso provide a motivating argument for convergence to the true hypergradient,\nand perform tractable gradient-based optimisation of independent learning rates\nfor each model parameter. Our method performs competitively from varied random\nhyperparameter initialisations on several UCI datasets and Fashion-MNIST (using\na one-layer MLP), Penn Treebank (using an LSTM) and CIFAR-10 (using a\nResNet-18), in time only 2-3x greater than vanilla training.\n