What was built and released
The arXiv CS.CL paper describes ufakzeka-1, a 151M-parameter (182M with embeddings) decoder-only Turkish model pretrained from scratch on 13.5B tokens of openly licensed text and instruction-tuned for chat, at a total cost of about $286 in cloud GPU, API and notebook time. Components include a Turkish byte-level tokenizer (1.77 tokens per word), a three-stage pretraining schedule, and a post-training mixture of openly licensed and generated data. Evaluation covered release gates, a rule-checked sweep of 5,508 conversations, judged conversations, and hand tests, all with prompts held out from training and enforced by decontamination in the data build plus a checked-in invariant script run before each build. Weights, the data recipe, evaluation code, and the spend ledger are released under Apache-2.0. [1]
Why the paper is worth reading
The authors state the contribution is not the model's capability, which they say is what a model this size can be expected to have, but the record of building and measuring it. The three reported findings — a safety gate inflated by training on its own questions, training-seed variance matching the spread across recipes, and long-context identity tracking and multi-turn arithmetic unmoved by data changes — are framed as transferable lessons for other small-model efforts, with the model-size interpretation flagged by the authors as something the next, larger model will test. [1]
Sources
- ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch
ArXiv CS.CL (Computation and Language) · Reporting ·