The POS tagging task is a sequence tagging task, where the goal is to predict the correct part-of-speech for each token in a sentence. For training data, we use the Gimpel dataset from [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0323064#pone.0323064.ref022" target="_blank">22</a>] with the crowd-sourced labels provided by [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0323064#pone.0323064.ref023" target="_blank">23</a>] mapped to the universal POS tag set in [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0323064#pone.0323064.ref024" target="_blank">24</a>]. The dataset consists of 1000 tweets (17,503 tokens) labeled with Universal POS tags and annotated by 177 annotators. Each token received at least 5 annotations. The IAA is 0.725 and the average annotator accuracy with respect to the gold labels is 67.81%. We use the publicly available sample of the Penn Treebank POS dataset [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0323064#pone.0323064.ref025" target="_blank">25</a>] accessed from NLTK [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0323064#pone.0323064.ref026" target="_blank">26</a>] as our out-of-domain test set, which consists of 3,914 sentences from Wall Street Journal articles (100,676 tokens). Distribution shift on this task is based on the data distribution (source: tweets, target: news). (PDF)