In this paper, we present our systems submitted to SemEval-2021 Task 1 on lexical complexity prediction (Shardlow et al., 2021a). The aim of this shared task was to create systems able to predict the lexical complexity of word tokens and bigram multiword expressions within a given sentence context, a continuous value indicating the difficulty in understanding a respective utterance. Our approach relies on gradient boosted regression tree ensembles fitted using a heterogeneous feature set combining linguistic features, static and contextualized word embeddings, psycholinguistic norm lexica, WordNet, word-and character bigram frequencies and inclusion in word lists to create a model able to assign a word or multiword expression a context-dependent complexity score. We can show that especially contextualised string embeddings