BanglaLinguistRelated repository: Banglalinguist Abstract This data descriptor documents seven primary datasets created, collected, or curated for BanglaLinguist, an integrated Bangla (Bengali) natural language processing pipeline covering Banglish (Romanized Bangla) normalization, part-of-speech tagging, lemmatization, named entity recognition, and dependency parsing under a Paninian karaka annotation scheme. The seven resources are a dependency treebank, a Banglish-to-Bangla parallel corpus (own-collected portion), a Banglish spelling-variant dictionary, an English-loanword dictionary, two independently authored cross-domain evaluation sets, and a dually reviewed part-of-speech/dependency annotation pilot. We describe each dataset's collection and annotation methodology, report its size and format as documented in the associated code repository, and summarize the technical validation evidence available — including a formal inter-annotator-agreement study for the annotation pilot. We also disclose, rather than conceal, every point where the underlying evidence is incomplete: two datasets are not yet publicly accessible, several reported record counts conflict between the originating research paper and the repository's own documentation, and one dataset's own-collected portion has been withdrawn from public release due to an unresolved data-provenance question. This descriptor is intended to accompany, not replace, the full research paper and its code repository. 1. Background and Summary Bangla is spoken by over 230 million people, yet NLP tooling for it remains comparatively under-resourced next to high-resource languages, and informally Romanized Bangla (“Banglish”) adds a further normalization burden that most existing resources do not address. The BanglaLinguist project assembled a pipeline and a supporting set of datasets to study this problem directly, with a particular focus on cross-domain generalization: whether a system trained on one distribution of text performs as well on independently authored, differently sourced text. The seven datasets described here were built or curated specifically to support that study. Four are entirely original compositions or curations (the dependency treebank, the two cross-domain evaluation sets, and the annotation pilot derived from them); two are dictionaries mined and hand-curated from the project's own data; and one — the Banglish parallel-pairs corpus — has a more complex, mixed provenance that this descriptor treats with particular care. This descriptor's purpose is narrow: to let a future user of these datasets understand what each one is, how it was built, how reliable the available evidence is, and what is and is not currently available for reuse. 2. Methods 2.1 KGP Dependency Treebank The treebank (8,468 sentences, 39,299 tokens, as of its most recent documented revision) carries gold part-of-speech, lemma, and karaka-scheme dependency annotations. Source sentences were composed fresh by a crowdsourced contributor pool to reflect conversational, daily-life, social-media-style, and news-style registers; the repository's own documentation states explicitly that these sentences were not transcribed or excerpted from real social media posts or news articles, which removes the copyright and platform-terms questions that affect the Banglish parallel-pairs corpus. Annotation was performed by two native Bangladeshi annotators — one a native Bangla speaker with strong dialect proficiency, the other a computer-science graduate with NLP annotation experience — following a shared guideline, with disagreements resolved through discussion and a final verification pass by the project supervisor. No formal inter-annotator-agreement statistic was computed for this particular annotation pass; this is disclosed as a limitation, not elided. 2.2 Banglish Parallel Pairs (own-collected portion) This resource (approximately 21,330–21,340 rows) pairs Romanized Banglish sentences with gold Bangla-script translations. Its source material is genuinely mixed: Common Crawl text, Bengali news-portal journalism, Wikipedia, public Facebook and Instagram posts, and sentences contributed directly by native speakers. Automatic filtering removed duplicate, incomplete, and encoding-damaged rows, and a subsequent manual pass reviewed the remainder for grammatical correctness and translation accuracy. Because a later, row-level provenance check found that no per-row source label survives in the current file, it is not possible to isolate the social-media-derived rows from the rest, and this own-collected portion has accordingly been withdrawn from public release (see Section 6). 2.3 Banglish Variant Dictionary and English-Loanword Dictionary The variant dictionary (reported as 940 entries in the originating paper and 941 in the repository's own documentation — an unresolved discrepancy) maps Romanized-spelling variants to their standard Bangla-script form. The English-loanword dictionary (reported as 2,129 entries in the paper and 2,328 in the repository documentation for the same development round — likewise unresolved) was mined by aligning the cross-domain gold translations word-for-word against their Banglish source and manually classifying each resulting pair as a phonetic borrowing (e.g., busy→বিজি) or a semantic translation (e.g., job→চাকরি). Both dictionaries are curated, tabular lookup resources rather than scraped corpora. 2.4 Cross-Domain Evaluation Set v1 and Validation Set v2 Set v1 (250 sentences across News, Social, Banglish, and Mixed/code-switched domains) and set v2 (152 sentences after deduplication) were both manually written rather than sampled from an existing corpus, each paired with a gold Bangla-script translation. Set v2 was written independently of set v1, specifically to provide a held-out generalization check uncontaminated by the dictionary-mining process that drew on set v1. The repository's own documentation flags one open item here: the precise authorship and sourcing method for these sentences — whether any were adapted from real social-media posts rather than authored fresh — is not yet formally documented, and this descriptor does not claim that question is resolved. 2.5 POS/Dependency Annotation Pilot This 120-sentence, 711-token pilot reuses sentences from sets v1 and v2 and carries a full part-of-speech and dependency annotation produced by an AI-assisted first draft, then fully re-annotated from scratch by two independent native-Bengali-speaking reviewers. An earlier draft version of the underlying source sentences was found, on pre-release audit, to contain real identifiable content — a named individual in the context of unverified abuse allegations, and a misidentified politician — and those sentences were regenerated before the current annotation was produced. 3. Data Records Table 1 summarizes the seven datasets. Items 3–7 are hosted in the project's GitHub repository; items 1–2 are excluded from the repository via.gitignore and were not located anywhere in the authoring workspace during this audit, so they are documented here but not currently retrievable. Dataset Format / Size Records KGP Dependency Treebank TSV; size not verified 8,468 sent / 39,299 tok Banglish Parallel Pairs (own) TSV; size not verified ~21,330–21,340 rows Banglish Variant Dictionary TSV; 23,789 bytes 940 (paper) / 941 (repo doc) English-Loanword Dictionary TSV; 137,820 bytes 2,129 (paper) / 2,328 (repo doc) Cross-Domain Evaluation Set v1 TSV; 33,551 bytes 250 sentences Cross-Domain Validation Set v2 TSV; 19,287 bytes 152 sentences (post-dedup) POS/Dependency Annotation Pilot TSV; 41,810 bytes 120 sent / 711 tok 4. Technical Validation The strongest available validation evidence concerns the POS/dependency annotation pilot: a second, fully independent reviewer re-annotated all 711 tokens from scratch, and formal inter-annotator agreement was computed against the first reviewer's completed pass, giving Cohen's kappa of 0.992 for part-of-speech tags and 0.820 for dependency relations, with 90.58% raw agreement on head attachment. All 117 token-level disagreements were adjudicated by hand with a written justification, and applying those decisions changed only 9 of 711 final labels. No comparable formal agreement statistic has yet been computed for the KGP treebank's or the Banglish parallel pairs' original annotation passes; both are disclosed here as not yet formally validated in that sense. The cross-domain evaluation sets passed a human review that found and corrected a duplication error (152 of an original 260 rows in set v2 were independent after 108 exact duplicates of set v1 were removed), and a separate pre-release content-safety audit on the annotation pilot's source sentences, described in Section 2.5. 5. Usage Notes All available files are plain tab-separated text and can be loaded with any standard TSV reader. Items 3–7 are suitable for direct reuse in Banglish normalization, dictionary-based code-switch handling, and cross-domain Bangla NLP evaluation. Item 2 should not be treated as reusable until its release status changes (Section 6); item 1 should be requested directly from the corresponding author pending a public release. 6. Data Availability and Limitations Items 3–7 are available now from the project's GitHub repository (https://github.com/tonoysheikh/banglalinguist). Items 1 and 2 are not currently available anywhere this audit could locate. Three record-count figures remain unresolved between the originating paper and the repository's own dated documentation: the English-loanword dictionary (2,129 vs. 2,328 entries), the Banglish variant dictionary (940 vs. 941 entries), and the Banglish parallel-pairs total, where the repository's most recent dated entry (62,115 combined pairs) postdates and differs from the paper's reported figure (63,096).