← research log

research log · 001 · 2026-08-02

keep the
diacritics.

Stripping Romanian's five marked letters makes a language model look better on the metric you'd naturally compute — and 57% worse at the language itself. Here is the whole study, with every model on both sides of the experiment available to download.

01 the decision every Romanian model makes

Romanian is written with five diacritic-bearing letters — ă â î ș ț — and the marks carry meaning: fată is a girl, față is a face. Yet Romanians routinely type without them, so anyone building a Romanian model faces a real choice: keep the diacritics and pay for them in vocabulary, or strip them and collapse distinct words into the same string.

LLMic — the only from-scratch Romanian foundation model at serious scale — chose to strip them, citing preliminary experiments where the characters “degraded model performance.” We wanted to know whether that trade-off is real. So we measured it.

02 the trap

We trained matched models on the same Romanian text under different diacritic treatments — same token budget, same seed, predictions registered in version control before any run. The result reproduces LLMic's finding and reverses its conclusion:

−2.5%stripping wins when each model is scored on its own view of the text — the metric a practitioner naturally computes
+57.1%stripping loses when both models are scored on the same correct Romanian — the metric that reflects the task

Same models, two evaluations, opposite verdicts. The stripped model scores best of any arm on the easy metric and worst by a wide margin on the honest one, because it has to fall back to raw bytes on every ș ț ă â î it meets. The measurement was never wrong — the metric was chosen after the result instead of before it.

03 scale does not save you

The obvious objection: maybe bigger models disambiguate si-versus-și from context, and the penalty fades with capacity. We pre-registered that prediction — 60% confident the penalty would shrink — and repeated the comparison at three sizes with the token budget pinned.

bits-per-byte penalty of the stripped model vs its diacritic-keeping twin, on identical correct Romanian
0%20%40%60%+54.3%73M+57.1%126M+59.8%286M
view data
paramsnormalizedstrippedpenalty
73M1.140611.75965+54.3%
126M1.059991.66477+57.1%
286M0.978191.56298+59.8%

The prediction was wrong, in the opposite direction: the penalty rises monotonically across a 3.9× span of model size. We don't claim a growth trend — the steps sit inside our pre-declared noise threshold — but the claim the experiment was run for stands: the penalty does not diminish with scale. The mechanism is tokenizer arithmetic, not a knowledge gap that capacity could fill.

04 two encodings, one letter

Along the way we found something nobody seems to have written down. Romanian ș and ț exist in two incompatible Unicode forms: the correct comma-below (U+0219, U+021B) and a legacy cedilla (U+015F, U+0163) left over from pre-Unicode-3.0 codepages. Sampling 73,580 documents across eleven years of CommonCrawl: 59.2% of Romanian web documents mix both encodings in the same page.

legacy cedilla share of Romanian ș/ț on the web — the signature of an encoding being phased out
0%20%40%60%59.6%201544.3%201824.9%202117.1%2024
view data
year2015201820212024
cedilla share59.6%44.3%24.9%17.1%

To an unnormalized tokenizer these are different tokens, so the training signal for one letter is split in two — and the model trained on raw text inherited the mess, writing ţară with a cedilla in one completion and vacanță with a comma in another. The two arms that never saw the legacy form never produce it. Folding cedilla to comma-below is lossless and costs nothing. Do it unconditionally.

05 what stripping destroys

Grouping the corpus vocabulary by its stripped form: 9.2% of word types merge, and because the collisions concentrate in the highest-frequency function words, 32.7% of running Romanian text becomes ambiguous. The words a model needs most are exactly the ones that collapse:

06 the fix is asymmetric

There is a legitimate worry behind stripping: real users type without diacritics, and a model trained only on pristine text becomes weirdly fragile. We hit this ourselves — the same fine-tuned model that answers „Cum te cheamă?” perfectly falls apart on the unaccented version:

Cum te cheamă?→ Sunt rost, un model lingvistic român–englez…

cum te cheama?→ Nu, nu poți cheama în mod constant…

The failure is diagnostic: unaccented esti gets matched to estima — the model knows the standard spelling so well the informal one is out of distribution. The resolution is not to strip the training data; it is to strip a fraction of the questions in instruction tuning while keeping every answer in correct orthography. In our runs this fixed the failure at zero measurable cost.

Keep diacritics in what the model learns and produces. Accept their absence in what it reads from users.

07 honest limits

08 check our work

Every model on both sides of this experiment is public — both twins at every scale, the unnormalized arm, the English control, and all three tokenizers. Cross-evaluate them yourself: