daari (ದಾರಿ — "path")

Compile your frozen LLM into a zero-matmul speculative drafter — no training, ~15 minutes, one GPU.

daari clusters your teacher model's own hidden states over your corpus into a finite map of semantic nodes, records the observed dynamics as lookup tables, and drafts tokens by walking that map — a 32-way argmax plus a binary search per token, no matrix multiplication. The teacher verifies drafts in single forward passes and the walker is re-anchored onto the teacher's true hidden state every round, so drift is structurally impossible and output is the teacher's own greedy stream.

On a production SMS-classification workload (Qwen3-4B teacher, NVIDIA GB10), a 9 MB compiled graph delivers:

metric	value
wall-clock speedup	4.33× (17.4 → 75.2 tok/s)
tokens per teacher forward	4.16
output	teacher-identical (greedy), up to teacher-internal bf16 ties
artifact	9 MB .npz · compiled in 35 s from 3.56M states

Full method, forensics, and honest prior-art positioning: paper/daari-paper.md.

Compile your own daari

pip install -r requirements.txt

# 1. extract teacher states over YOUR data (~minutes, one forward per sequence)
python -m daari.extract --teacher your-org/your-model \
    --data train.jsonl --workdir ./work

# 2. compile the graph (~seconds)
python -m daari.compile --workdir ./work

# 3. benchmark against vanilla greedy on held-out data
python -m daari.spec_decode --teacher your-org/your-model \
    --graph ./work/graph.npz --data test.jsonl --n 30

train.jsonl rows are either chat ({"messages": [...]}, rendered with your model's chat template) or plain text ({"text": "..."}).

Use it in code

from daari.spec_decode import SpecEngine

eng = SpecEngine("your-org/your-model", "work/graph.npz")
text, stats = eng.generate_chat(
    [{"role": "user", "content": "sender: XYZ\nmessage: ..."}])
print(stats)   # {'tokens': ..., 'calls': ..., 'tokens_per_call': ...}

Watch it think

python -m daari.annotate --teacher your-org/your-model \
    --graph ./work/graph.npz --data test.jsonl --idx 0

Prints every round: the graph's drafted path (node ids + edge probabilities), the teacher's ✓/✗ verdicts, the bonus token, and the re-anchor — the whole negotiation, live:

── round  3 ──────────────────────────────────────────────
  graph : n5692─1.00→' null' n4356─1.00→',' … n6520─0.62→'1'
  teacher: ✓✓✓✓✓✓✓  accepted 7/7 — all agreed
  bonus : '0'   → 8 tokens this round, 1 forward
  re-anchor → n5092

When does this work well?

daari shines where your outputs have structure: classifiers with JSON output, extractors, form-fillers, routers, agent tool-calls — any workload whose sublanguage is much smaller than open text. Diagnostic: after compiling, check compile_stats.json — high mean top-edge mass (ours: 0.86) and high path reuse (ours: 43×) predict high acceptance. Free-form creative text will draft poorly (the system then gracefully degrades to ≈vanilla speed — verification means it is never worse than the teacher alone).

Best hardware fit: memory-bandwidth-bound, batch-1 decoding (unified-memory devices like DGX Spark / Apple Silicon, or any single-stream serving), where a verify pass over γ+1 tokens costs ≈ one token's weight-streaming and drafted tokens ride free.

What daari is not

• Not a LoRA — nothing is trained; the graph is compiled from a frozen teacher (minutes, deterministic).
• Not a standalone model — the graph is teacher-locked (it lives in that model's hidden geometry) and domain-bound. Standalone walking nails rigid templates and provably degenerates in open text; the drafter deployment is the honest configuration.
• Not a hallucination-safe oracle — quality equals the teacher's, no more.

Example: sms-shield

examples/sms_shield/ contains the compiled graph (9 MB), compile stats, and the 30-prompt benchmark results from the paper. Note: the graph only functions with its exact teacher checkpoint (a merged Qwen3-4B LoRA for Indian SMS scam classification).

Prior art, honestly

Every ingredient here exists in prior work: lexical automaton drafters (SAM-Decoding, SuffixDecoding), drafting from hidden states (EAGLE-1/2/3), verifier-state-keyed retrieval drafting (Oilbird, SENSE — continuous forms), LLM→discrete-surrogate distillation (Ctrl-G). daari's contribution is the composition: post-hoc discretization of a frozen teacher's states into an executable automaton, drafted with zero matmuls and re-anchored per round. Full comparison table and citations in the paper (§2).

Citation

@misc{daari2026,
  title  = {daari: Compiling a Frozen Transformer into a Zero-Matmul
            Speculative Drafter},
  author = {Santosh},
  year   = {2026},
  note   = {Ganmoor AI Labs},
}