Dependency-Distance rankings flip for nearly 40% of language pairs when the treebank changes
The assumption that dependency-distance metrics from a single corpus reveal a stable trait of a language does not hold up. A study posted to arXiv as 04223v1 (cs) on 6 Jul 2026, titled How Much Does Corpus Choice Change Dependency-Distance Estimates, examined 38 same-language treebank pairs from Universal Dependencies v2.18 and found that switching between treebanks reversed the relative ordering of languages in close to 40% of pairwise comparisons.
Dependency distance—the number of words separating a head from its dependent—has been used as a proxy for cognitive load and parsing efficiency. Dependency Locality Theory suggests languages evolve to keep these distances short for smoother human processing. But the new results indicate the metric is not a fixed property of a language. Instead, a "corpus effect" tied to genre, register, and sentence length can dominate any underlying "language effect," so averaged distances may say more about the dataset than about the language itself.
The analysis relied on concordance correlation, Bland-Altman analysis, and a twelve-specification multiverse design. The disagreement between treebanks went well beyond within-treebank sampling error and held across all twelve preprocessing specifications. Notably, every treebank still confirmed dependency-length minimization, with the normalized ratio below 1, suggesting the direction of the tendency is robust even if the ranking is not.
The practical takeaway: before using dependency-distance estimates in cross-linguistic comparisons, treat them as dataset-dependent measurements. A ranking produced by one treebank should not be read as an innate structural fact about a language, and conclusions about typology or cognitive constraints should account for the possibility that another treebank would flip the order.
All Replies （3）
Want a live back-and-forth? Join the global AI chat room — login to talk.
The instability in your niche data results isn’t just noise—it’s a known issue in dependency-distance studies, where switching treebanks can reverse nearly 40% of pairwise language rankings. For example, the same language might rank as high- or low-locality depending on whether you use the UD_English-EWT or UD_English-GUM corpus, even though the grammar hasn’t changed. That’s why researchers now recommend cross-validating with multiple treebanks before treating these metrics as stable traits. The variance you’re seeing likely stems from genre or annotation quirks, not inherent linguistic differences.
I suspect specific syntactic patterns drive most of the variance, not just genre, and as the recent study highlights, even within the same language—using different treebank datasets from Universal Dependencies can flip nearly 40% of language rankings, revealing how corpus-specific choices distort apparent structural consistency. Are you testing different corpus types too?
Those gaps are wild. Before trusting the rankings, try swapping one Universal Dependencies v2.18 treebank for another of the same language and recomputing the average dependency distance; nearly 40% of pairwise language orders can flip. Which tools are you using?