Do Generative Large Language Models Close the Language Gap in Multilingual Discourse Classification? An Evaluation Across Three European Languages

Abstract

Uneven language support and resource asymmetries remain a central challenge for applying large language models to social-science text analysis. This study evaluates whether generative models narrow that gap on two multilingual classification tasks – xenophobic and anti-government discourse – using 6,000 social media posts in Danish, Swedish, and German. It compares 12 base models evaluated under supervised fine-tuning and zero-/few-shot prompting (33 model-setting configurations). The support gap persists – model configurations applied to German (or English translations) reliably outperform those for Danish and Swedish; fine-tuned encoders set the ceiling (macro-F1 ≈ 0.85 in German), while few-shot prompting yields only modest gains over zero-shot inference. A consensus analysis of per-item agreement shows higher model-gold alignment for xenophobia than anti-government (high-agreement band: 76.6% vs 52.3% of items), with residual errors concentrated in mediated discourse and irony/euphemism. Methodologically, the study introduces a consensus-based validation approach that groups items by model agreement and employs mixed-effects modeling to identify systematic failure, providing a reproducible bridge between quantitative reliability and qualitative error analysis. Overall, uneven language support governs reliability in discourse-level classification, challenging claims that frontier decoders close multilingual measurement gaps and underscoring the need for evaluation infrastructure for smaller and less-supported languages in communication research.