PTNews Corpus

Publication

AuthorsDavide Nunes

Year2020

JournalZenodo (CERN European Organization for Nuclear Research)

Abstract

The PTNews Corpus is a collection of over 19 million tokens extracted from 10 years of political news articles (in Portuguese) from the Portuguese newspaper PÚBLICO. The corpus is available under the Creative Commons Attribution-NonCommercial-ShareAlike Licence. The material contained on the PTNews Corpus is © 2010-2020 PÚBLICO Comunicação Social SA. The corpus sizes between the preprocessed version of Penn Treebank (PTB) and WikiText-103. Similarly to WikiText, PTNews has a larger vocabulary than PTB and retains the original case, punctuation and numbers. This corpus contains over 31000 publicly available full articles which makes it well suited for models that can take advantage of long-term dependencies. The corpus is available as a word-level collection of articles in two version: the first (ptnews_origin) contains a single file with all the articles in the form: title, URL, date, body; the second, contains only the title and body of the news articles and it is split into train, test, validation sets. In this processed version, the words with less than 3 occurrences are mapped to the <unk> token. Each sentence in an article body occupies a single line of the dataset and the end of paragraph is marked with the <eop> tag at the end of a sentence. Portuguese words resulting from contractions like "desta", ou "nesta" are separated into "d", "esta", "n", "esta", respectively. Sample article: <pre><code class="language-bash">Carlos César: Cavaco " cansado e sem entusiasmo " quis afastar responsabilidades sobre a crise https://publico.pt/2010/06/10/politica/noticia/carlos-cesar-cavaco-cansado-e-sem-entusiasmo-quis-afastar-responsabilidades-sobre-a-crise-1441369 2010-06-10 15:38:00 O presidente do Governo Regional dos Açores, Carlos César, considerou hoje que Cavaco Silva esteve " cansado e sem entusiasmo " no discurso do Dia de Portugal, onde afastou responsabilidades sobre a actual crise. <eop> " O país ouviu um Presidente cansado e sem entusiasmo, que andou às voltas com os papéis para dizer que não tinha nada a ver com as razões da crise ", afirmou Carlos César, num comentário à Lusa sobre o discurso do Presidente da República na cerimónia oficial do 10 de Junho, realizada em Faro. <eop> Carlos César considerou, no entanto, " positivo " que Cavaco Silva tenha feito " um discurso alinhado com um tema recorrente na apreciação do momento que vivemos, o da coesão e da corresponsabilização ". <eop> No mesmo sentido, manifestou concordância com o apelo que Cavaco Silva fez " à responsabilidade dos empregadores e empregados ", mas deixou um alerta relativamente à referência do Presidente da República à necessidade de " limpar Portugal ". <eop> Para Carlos César, se essa referência " for despida de conteúdo institucional útil, tratou-se de mais um discurso que se perderá na babugem política d aquilo que Cavaco Silva entendeu recordar como o ' rectângulo ' ". <eop></code></pre> Reporting Results If you wish to report results or other resources obtained on the PTNews contact Davide Nunes with the following information: Task: e.g. Language Modelling, Semantic Similarity, etc; Publication URL: url to published article or preprint; Type of Model: LSTM Neural Network, n-grams, GloVe vectors, etc; Evaluation Metrics: e.g. validation and testing perplexities in the case of language modelling. They will be displayed here Preprocessed Corpus Statistics articles: 31.919 articles by split: train: 25.537 test: 3.191 val: 3.191 unique tokens: 68.318 unique OoV Tokens: 76.157 total tokens: 19.021.661 total OoV tokens: 95.043 OoV rate: 0.5% tokens by split: train: 15.242.995 test: 1.895.184 val: 1.883.482 Contact Information If you have questions about the corpus or want to report benchmark results, contact Davide Nunes.