Morpheus-TR-50K

Morpheus is a neural morpheme-aware tokenizer and word embedder for Turkish — an agglutinative language whose semantic content is densely packed into productive suffix chains. It combines unsupervised morphological supervision (Morfessor) with self-supervised objectives (SGNS, contrastive, MLM) to learn morphologically aligned segmentations. Because it is neural, the same forward pass that tokenizes also yields a structured word embedding — so Morpheus is a tokenizer and an embedding model at once.

evlerimizdekiler  →  ev | ler | imiz | de | ki | ler
                     root  PL   POSS   LOC  REL  PL
                    ("the ones in our houses")

Segmentation only groups characters, never rewrites them; with a lossless pre-tokenizer and byte fallback, decode(encode(x)) == x holds by construction for any input text — case, punctuation, whitespace, code, and emoji included.

Headline Results

Morpheus is exactly reversible on raw text and, among reversible tokenizers, the only one whose tokens are morphemes. All baselines below preserve case and were trained on the same corpus.

Tokenizer	Inflected words (30,204)	Raw corpus lines (5,000)	Stress set (14)
Morpheus	100%	100%	14/14
Byte-level BPE (GPT-2 style)	100%	100%	14/14
BPE / Unigram (SentencePiece, NFKC)	100%	98.7%	7/14
WordPiece (cased)	100%	24.2%	3/14
TurkishTokenizer (rule-based)	95.4%	0%	0/14

Byte-level BPE is the only other exactly reversible tokenizer, so the comparison that matters is between the two:

	Morpheus	Byte-level BPE
Exact decode (words / raw text / stress)	100 / 100 / 14	100 / 100 / 14
MorphScore macro-F1 ↑ (UD gold)	0.61	0.44
Rare roots segmented exactly ↑	70%	10%
TR-MMLU pure tokens (unique) ↑	57.1%	32.1%
BPC ↓ (58M GPT, equal 10K steps, single seed)	1.451	1.411
Characters per token ↑	3.81	4.97
Generation, chars/s at batch 1 ↑	474	874
Tokenizer artifact ↓	29.6 MB	4.6 MB
Word embedding from the same model	✓	—

The trade-off: Morpheus's morpheme units make the same text 30% longer as a token sequence than byte-level BPE, and in a controlled language-modeling probe (58M model, equal compute, single seed) its token stream reaches a 2.8% higher BPC — under an equal-token budget Morpheus also sees 23% less text. What it offers instead is the combination of exact reversibility with morphologically aligned units, plus a root-centric word embedding. Choose Morpheus when surface fidelity and morpheme-level units both matter; byte-level BPE when sequence length is the binding constraint. Integrating Morpheus into a full Turkish LLM is follow-up work.

Morpheus as a word embedder

The same forward pass that tokenizes emits a 320-dim word embedding. Evaluated frozen against BERTurk (768-d) and the multilingual retriever BGE-M3 (1024-d) — both encoding isolated words — the picture splits by task character: Morpheus dominates lexical / root-level tasks, the heavier contextual encoders lead on context- and inflection-dependent tasks.

Task	Morpheus (320)	BERTurk (768)	BGE-M3 (1024)
Root-family retrieval (MAP ↑, 120 gold families)	0.89	0.38	0.72
Same-root verification (ROC-AUC ↑)	0.999	0.794	0.951
Number probing (acc ↑; majority class 0.52)	0.59	0.95	0.91
Case probing (acc ↑; majority class 0.20)	0.22	0.89	0.81
WikiANN-tr NER (macro-F1 ↑)	0.48	0.79	0.76

This is a deliberate, architectural trade-off: the root-identity contrastive objective pulls a root's inflections together — sharpening root geometry (hence the retrieval/verification wins) while collapsing the inflectional contrasts a probe reads — and the static per-word vector lacks the sentence context NER needs. Morpheus is therefore complementary to contextual encoders: well suited to the lexical index of a multi-vector RAG system, paired with a dense semantic encoder for context.

Model Details

• Developed by: Tolga Şakar
• Model type: Neural morpheme-aware tokenizer + word embedder (CharEncoder → BoundaryDetector → SegmentEncoder)
• Language(s): Turkish (tr)
• License: MIT
• Trained from: Scratch (no base model)
• Vocabulary size: 50,000 tokens — morpheme-level segments, plus 256 byte-fallback tokens and whitespace tokens (<NL>, <TAB>, <CR>) reserved inside the budget
• Word embedding dimension: 320

Files in this repository

File	Size	Purpose
morpheus_50k/vocab.json, morpheus_50k/tokenizer_config.json	small	Tokenizer vocabulary and settings (tokenizer v2: lossless pre-tokenizer + byte fallback)
turkish_morpheus_a100_v3_inference.pt	~30 MB	Inference checkpoint: model weights + config as plain values. Loaded by default.
turkish_morpheus_a100_v3_best.pt	~600 MB	Full training checkpoint (adds optimizer, SGNS embeddings, MLM head). Needed only to resume training or run the evaluation suite.

Repository

• Code: TurkishMorpheus
• Corpus (train/test splits): lonewolflab/Morpheus-TR-corpus
• Corpus collection tooling: CorpusCollector
• Paper: Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
• Demo: morpheus-tr-demo

Usage

Morpheus is a neural tokenizer — it needs the vocabulary and a model checkpoint. Standard AutoTokenizer.from_pretrained() does not apply.

Install

git clone https://github.com/lonewolf-rd/TurkishMorpheus.git
cd TurkishMorpheus
pip install -e .

Load and tokenize

from_pretrained downloads morpheus_50k/ and the ~30 MB inference checkpoint from this repo (falling back to the full checkpoint if the inference file is absent) and builds the model on GPU when available. Pass a local directory instead of the repo id to load offline, or device="cpu" to force CPU.

from src.model_development.tokenization.morpheus_tokenizer import MorpheusTokenizer

tokenizer = MorpheusTokenizer.from_pretrained("lonewolflab/Morpheus-TR-50K")

text = "Evlerimizdekiler, kitapçısı muvaffakiyetsizleştiriciler."
tokens = tokenizer.tokenize(text)
ids = tokenizer.encode(text, add_special_tokens=False)
decoded = tokenizer.decode(ids)

print(tokens)
print(decoded == text)  # True for any input text

To tokenize a large corpus, segment its distinct words in large batches first; later encode calls then only read the cache:

lines = open("corpus.txt", encoding="utf-8").read().splitlines()
tokenizer.warm_cache(lines, batch_size=2048)
ids = [tokenizer.encode(line, add_special_tokens=False) for line in lines]

Word embeddings

import torch

# Morpheus produces a 320-dim embedding per word from the same forward pass
morpheus, device = tokenizer.model, tokenizer.device
words = ["kitap", "kitaplar"]
ids, flags, lens = [], [], []
for w in words:
    i, f, rl = tokenizer.helper.word_to_char_ids(w, max_len=32)
    ids.append(i); flags.append(f); lens.append(rl)
with torch.no_grad():
    out = morpheus(
        char_ids=torch.tensor(ids, device=device),
        case_flags=torch.tensor(flags, device=device),
        real_lengths=torch.tensor(lens, device=device),
    )
emb = out["word_embeddings"]   # (2, 320)

Note: the inference checkpoint holds only tensors and plain values and loads with torch.load(..., weights_only=True). The full training checkpoint stores a pickled config object and needs weights_only=False — only load it from sources you trust.

Architecture

char_ids, case_flags  (B=batch, L=32 max word chars)
        │
        ▼
   CharEncoder              Multi-scale CNN + 3 × RoPE self-attention
        │                   Output: (B, L, 320) context-aware char vectors
        ▼
  BoundaryDetector          4 × RoPE attention with adjacent-pair scoring
        │                   Deep-supervised aux loss vs Morfessor labels
        │                   Output: boundary_probs ∈ [0,1] of shape (B, L−1)
        ▼
  SegmentEncoder            Poisson-binomial DP → soft segment membership
        │                   Char-level attention pooling per segment
        │                   Mean over valid segments + FFN + LayerNorm
        ▼
   word_embedding ∈ ℝ³²⁰

Soft segmentation via Poisson-binomial dynamic programming: given boundary probabilities p_i ∈ [0,1] between adjacent characters, the probability that character i belongs to segment k is computed via:

f[i, k] = f[i−1, k] · (1 − p_i) + f[i−1, k−1] · p_i

This gives a differentiable membership matrix that converges to one-hot segment assignments as p_i → {0,1}, recovering hard segmentation at inference without an architectural switch. Because the operator only groups characters (never rewrites them), segmentation is surface-preserving.

From segments to ids: text is pre-tokenized into words, single symbols, and whitespace without discarding anything; a single space is carried by a ▁ word-boundary marker, extra spaces and newlines/tabs/CRs get their own tokens. Each word is segmented by Morpheus; segments missing from the vocabulary fall back to single characters, and characters missing from the vocabulary to their UTF-8 bytes. Every step has an exact inverse, so decode(encode(x)) == x for any text.

Training Data

Trained on a large-scale monolingual Turkish corpus combining a multi-register author corpus with the full cleaned Turkish Wikipedia (~10 GB raw text), covering four registers:

• Ekşisözlük — informal/colloquial Turkish (rich morphological constructs)
• Dergipark — academic/formal Turkish (diverse derivational morphology)
• Turkish news sites — standard journalistic register
• Turkish Wikipedia — encyclopedic, broad vocabulary (aggressively cleaned/filtered + dedup)

The web-sourced registers were collected with the companion repository CorpusCollector, which documents source URLs, scraping protocol, rate-limiting, and per-source preprocessing for full reproducibility. Split: 95% train / 5% test, fixed seed; both splits are published as lonewolflab/Morpheus-TR-corpus. The checkpoint was trained on an earlier snapshot of this corpus; the vocabulary and all evaluations use the rebuilt snapshot.

Training Procedure

Hyperparameters

Parameter	Value
Epochs	10
Effective batch size	512 (256 × grad_accum 2)
Optimizer	AdamW
Learning rate	cosine-decayed
Char dim	320
Encoder / Detector layers	3 / 4
Attention heads	5
Max word len	32
Max segments per word	12
Dropout	0.1
Mixed precision	TF32 (matmul + cuDNN); loss in FP32; AMP off

Loss

L = w_aux · L_aux + w_sgns · L_sgns + w_ctr · L_contrastive + w_mlm · L_mlm

Component	Role	Weight schedule
L_aux	Boundary BCE + count MSE vs Morfessor (root-corrected, deep-supervised)	Decays 0.50 → 0.08 over 10 epochs
L_sgns	Skip-gram with 16 negatives, ±6 window, 120K context vocab	Constant 0.7
L_ctr	InfoNCE on root identity, temperature 0.10	Constant 0.3
L_mlm	Char autoregressive reconstruction of masked words (20% mask rate)	Constant 1.0

The aux schedule realizes a curriculum: early epochs anchor on Morfessor; as it decays, distributional signals (SGNS, MLM) take over to shape semantic geometry, so the model becomes teacher-free and generalizes to OOV.

Hardware

• Single NVIDIA A100 80GB, ~30 min/epoch × 10 epochs ≈ 5 hours total.

Evaluation

All baselines preserve case: BPE and Unigram are SentencePiece with nmt_nfkc, byte-level BPE is GPT-2 style with no normalizer, WordPiece is cased (NFC only); the Morfessor baseline is retrained on the same corpus.

Reversibility (the generation gate)

See Headline Results. BPE and Unigram lose NFKC-normalized characters (²→2, …→...) and repeated or non-breaking spaces; WordPiece loses spacing around punctuation; TurkishTokenizer canonicalizes suffixes (saatlerde → saatlarda) and inserts spaces.

Gold morphology — MorphScore (UD_Turkish-Kenet, 30K words) and SIGMORPHON 2022 (856 words)

Model	MorphScore macro-F1	SIGMORPHON lemma-prefix	SIGMORPHON root-in-segments
TurkishTokenizer (not reversible)	0.648	0.711	0.633
Morpheus	0.608	0.762	0.481
Morfessor	0.550	0.708	0.349
Byte-level BPE	0.441	0.617	0.167
Unigram / BPE / WordPiece	0.35–0.36	0.59–0.61	0.42–0.43

Surface fidelity by phenomenon (50 curated words, len% / exact%)

Phenomenon (n)	Morpheus	Morfessor	Best subword	TurkishTokenizer (not reversible)
Rare root (20)	70 / 70	40 / 40	45 / 45	90 / 20
Derivation (10)	30 / 30	40 / 40	20 / 20	80 / 10
Consonant softening (7)	14 / 14	0 / 0	0 / 0	100 / 0
Vowel drop (6)	17 / 17	33 / 33	0 / 0	0 / 0
Loanword harmony exception (7)	0 / 0	0 / 0	0 / 0	86 / 0

Overall exact%: Morpheus 38, Morfessor 28, WordPiece 22, Unigram 18, BPE 16, byte-level BPE 6, TurkishTokenizer 10.

Token validity — TR-MMLU (Kalbur validator)

Tokenizer	%Pure (unique)	%Pure (freq.-weighted)	Fertility (tokens/word)
TurkishTokenizer (not reversible)	65.5	78.2	1.98
Morpheus	57.1	76.5	1.90
Morfessor	46.5	75.9	1.82
Subword (BPE / Unigram / WordPiece / byte-level BPE)	32–33	48–51	1.48–1.56

Downstream language modeling — BPC (param-equalized 58M GPT, equal 10K steps, single seed)

Tokenizer	vocab	BPC ↓	Raw-text roundtrip	Chars / token
WordPiece	64K	1.380	24.2%	4.97
TurkishTokenizer	33K	1.407	0%	3.09
Byte-level BPE	64K	1.411 (best reversible)	100%	4.97
Unigram	64K	1.416	98.7%	4.59
BPE	64K	1.425	98.7%	4.60
Morfessor	50K	1.430	—	4.26
Morpheus	50K	1.451	100%	3.81

The two lowest values come from non-reversible tokenizers whose token streams omit information (WordPiece: spacing around punctuation; TurkishTokenizer: allomorph identity). Under an equal-token budget Morpheus covers 624M characters of training text against 815M for byte-level BPE. An earlier version of this card reported a lower Morpheus BPC (1.425); that run's pre-tokenizer dropped punctuation from Morpheus's token stream and the baselines folded case.

Inference / efficiency

Tokenizer	Gen chars/s (B=1 / B=32)	Peak GPU mem (B=32)	Encode (chars/s)	Decode (words/s)	Artifact
Morpheus	474 / 12,378	3,021 MB	1.51M	0.44M	29.6 MB
Byte-level BPE	874 / 21,787	3,724 MB	1.53M	0.62M	4.6 MB
BPE	827 / 20,672	3,724 MB	0.98M	0.32M	1.3 MB

The memory saving follows from the smaller vocabulary (50K vs 64K), not from morphology.

Intended Uses

Recommended

• Text pipelines where decoded output must match the input exactly (case, punctuation, whitespace, code, symbols)
• Morphological segmentation and analysis, especially of rare and technical vocabulary; linguistic research and educational tools
• Lexical / keyword retrieval, stemming, morphological dedup — and the lexical index of a multi-vector RAG system

Not recommended

• Throughput-bound generation (~1.8× slower than byte-level BPE at the tested scale)
• Settings where BPC at a fixed token budget is the deciding metric
• A drop-in replacement for a contextual encoder on NER or inflectional-feature tasks — pair Morpheus with BERTurk/BGE-M3 there
• Multilingual models (Turkish-specific by design)

Limitations

• Sequence length and BPC: 1.90 tokens/word (30% more tokens than byte-level BPE on the same text); BPC 2.8% higher at equal compute for a 58M model. Results are single-seed; equal-text, multi-seed, and larger-scale comparisons have not been run.
• Deployment: the tokenizer is a ~30 MB neural model that must run to segment unseen words (1.5M chars/s), not a lookup table.
• Character inventory: q, w, x and circumflexed vowels are not distinguished by the model, which weakens segmentation of loanwords and foreign names (reversibility is unaffected).
• Root-centric embedding: excels at lexical retrieval/verification but underperforms contextual encoders on number/case probing and NER — by architectural design, not a bug.
• Single language: trained for Turkish only.

Citation

@misc{sakar2026morpheus,
  title  = {Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish},
  author = {Şakar, Tolga},
  year   = {2026},
  note   = {arxiv.org/abs/2606.18717}
}

Acknowledgments

• Morfessor (Creutz & Lagus, 2002, 2007) — unsupervised morphological supervisor
• SentencePiece (Kudo & Richardson, 2018) and HuggingFace tokenizers — baseline implementations
• SIGMORPHON 2022 Turkish task — inflection gold standard
• BERTurk and BGE-M3 — embedding baselines
• The Turkish NLP community (BERTurk, TURNA, Zemberek, TRMorph) that motivated this study

License

MIT License. See LICENSE.

---

Repository: TurkishMorpheus · Author: Tolga Şakar