02 · the models
Small enough to study.
Open enough to check.
Four sizes, all trained from scratch on the same recipe — European foundation models in the literal sense, with no weights, no distillation and no tokenizer inherited from a larger model. The small series proved the tokenizer and the data mixture; rost-1b is what that recipe was spent on.
weights available1.384B
rost-1b
The trunk of the family: a bilingual base trained from scratch in two phases — broad web, QA and code first, then a Romanian-heavy anneal while the learning rate decays. 11.68B tokens in a single pass, no data repeated. Two instruction-tuned checkpoints share that base: v1, and v2 from a second fine-tuning experiment.
weights available286M
rost-286m
The strongest of the shipped series, and the model the recipe was proven on. The sft checkpoint is the best conversational model of the family.
weights available126M
rost-126m
The bilingual pair — pretrained on a 30:70 Romanian–English mixture. Understands both, and answers in the language you asked in.
weights available73M
rost-73m
The smallest of the family. Small enough to train honestly and inspect end to end — a working proof that the tokenizer, corpus, and recipe hold together.
not trained yet—
the specialists
Models branched from rost-1b that each do one thing well. Nothing is trained yet, and this card will say so until something is.
rost-1b, as published
Every number below appears on the model card. You can check each one before you trust it.
- parameters
- 1.384B
24 layers · 1,536 hidden · 12 heads- training
- 11.68B, single pass — no data repeated
8× H100 80GB, ~3.6 hours- context
- 4,096 tokens
vocab 32,768, purpose-built bilingual- licence
- CC-BY-NC-4.0 · non-commercial
Q4_K_M at 791 MiB, through to BF16 at 2.6 GiB
~91 tok/sRTX 5070transformers, bfloat16, ~3.3 GB resident489.7 tok/sRTX 5070llama.cpp, Q4_K_M60.6 tok/s12 CPU threadsllama.cpp, Q4_K_M77 tok/sApple M3llama.cpp, Q4_K_M OpenLLM-Ro, base checkpoint · 400 rows per task · zero-shot, normalised| task | score | chance |
|---|
| ro_hellaswag | 38.50 | 25.0 |
|---|
| ro_truthfulqa | 34.00 | 14.3 |
|---|
| ro_arc_challenge | 30.25 | 28.6 |
|---|
| ro_mmlu | 29.75 | 25.0 |
|---|
| ro_winogrande | 52.25 | 50.0 |
|---|
| mean | 36.95 | 28.6 |
|---|
Zero-shot, so not comparable to the OpenLLM-Ro leaderboard, which averages several few-shot settings. For scale rather than ranking: RoLlama2-7b-Base reports 42.05 on a four-task average against this model's 34.12 — from a model 5× larger trained on far more data.
rost-1b-instruct-v2 — what a second fine-tune actually bought
the same rost-1b base as v1 — no new pretraining, so no new knowledge. Both checkpoints were run in one session on one harness, so these are differences between the models rather than between two measurements.
measured, v1 against v2 | v1 | v2 |
|---|
| instruction following | 12/75 | 19/75 |
|---|
| replies that stop on their own | 15/75 | 33/75 |
|---|
| English likelihood benchmarks (mean) | 0.449 | 0.438 |
|---|
| Romanian, five OpenLLM-Ro tasks (mean) | 0.3685 | 0.3715 |
|---|
The Romanian difference is inside the ±2.4-point noise band at 400 rows a task: no measurable change. The English one is a small post-training regression — fine-tuning cannot add knowledge to a base it did not change, and this recipe cost a little likelihood while buying better behaviour. A sealed 44-case held-out set, written and encrypted before training and opened once afterwards, did not confirm a general improvement either.
Where it is weak: It writes worse Python than it appears to, it has no identity of its own where v1 will tell you it is rost, and one prompt in the repetition-stress set loops under every decoding setting tested. The repository therefore prescribes no sampling parameters, because none of the five tested passed the bar set before the results were read.
