RouteLLM reproduction β Llama-3.3-70B vs Llama-3.1-8B
A runnable reproduction of RouteLLM: Learning to Route LLMs with Preference Data (arXiv:2406.18665, Ong et al., ICLR 2025), built on the official lm-sys/RouteLLM implementation. A lightweight matrix-factorization router learns P(strong model wins | query) from pairwise preference battles, and routes each query to meta-llama/Llama-3.3-70B-Instruct (strong) or meta-llama/Llama-3.1-8B-Instruct (weak) to cut cost without proportional quality loss. The paper reports ~2-3.7x cost savings at matched quality with this method (for GPT-4 vs Mixtral).
Verified assets
| Role | HF id | Notes |
|---|---|---|
| D_arena (human battles) | lmarena-ai/arena-human-preference-55k |
57.5K rows, 9 cols |
| D_judge (GPT-4-judge battles) | routellm/gpt4_judge_battles |
109.1K rows |
| D_gold (MMLU golden battles) | routellm/mmlu_battles |
1.5K rows, MMLU validation split |
| Released MF checkpoint | routellm/mf_gpt4_augmented |
205K params, GPT-4/Mixtral pair |
| Threshold scores (upstream) | routellm/lmsys-arena-human-preference-55k-thresholds |
per-router scores, 57.5K rows |
| Strong model | meta-llama/Llama-3.3-70B-Instruct |
gated β accept the license first |
| Weak model | meta-llama/Llama-3.1-8B-Instruct |
gated β accept the license first |
| Embeddings (open) | Qwen/Qwen3-Embedding-0.6B |
1024-dim, Apache-2.0 |
| Benchmarks | openai/gsm8k (main/test, 1319), cais/mmlu (all/validation, 1531) |
8-shot / 5-shot as in the paper |
All three battle datasets share the schema id, model_a, model_b, prompt, response_a, response_b, winner_model_a, winner_model_b, winner_tie; prompt
is a JSON list of turns and routers are trained on the first turn (the same
convention as upstream routellm/calibrate_threshold.py).
Pipeline
prep_battles.py -> data/battles_{arena,judge,mmlu,all}.parquet (prompt, model_a, model_b, y)
embed_prompts.py -> data/embeddings_*.npy (Qwen3-0.6B, open, or OpenAI)
train_mf.py -> ckpt/mf_repro/ (~205K params, minutes on CPU)
calibrate.py -> ckpt/mf_repro/thresholds.json (quantile of scores)
eval_routed.py -> results/eval.json (accuracy / %strong / cost)
serve_router.py -> OpenAI-compatible routing proxy (optional)
pip install -r requirements.txt # pins routellm==0.2.0
python scripts/prep_battles.py
# Embeddings β the SAME backend must be used for training and all scoring.
python scripts/embed_prompts.py --battles data/battles_all.parquet \
--backend qwen3 --out data/embeddings_all.npy
python scripts/embed_prompts.py --battles data/battles_arena.parquet \
--backend qwen3 --out data/embeddings_arena.npy
python scripts/train_mf.py --battles data/battles_all.parquet \
--embeddings data/embeddings_all.npy --out-dir ckpt/mf_repro \
--push-to-hub <your-namespace>/mf_llama_repro
python scripts/calibrate.py --embeddings data/embeddings_arena.npy \
--checkpoint ckpt/mf_repro --text-dim 1024 --strong-pcts 0.5 0.25 0.8
# e.g. prints: For 50% strong-model calls, threshold = 0.xxxxx
Evaluation (two OpenAI-compatible endpoints, e.g. vLLM):
export STRONG_BASE_URL=http://localhost:8000/v1 STRONG_MODEL=meta-llama/Llama-3.3-70B-Instruct
export WEAK_BASE_URL=http://localhost:8001/v1 WEAK_MODEL=meta-llama/Llama-3.1-8B-Instruct
# smoke: 64 prompts, router policy only
python scripts/eval_routed.py --benchmark gsm8k --limit 64 --threshold <t> --out results/smoke.json
# full: router + always-strong + always-weak baselines
python scripts/eval_routed.py --benchmark gsm8k --threshold <t> --baselines \
--strong-price 0.60 --weak-price 0.03 --out results/gsm8k.json
python scripts/eval_routed.py --benchmark mmlu --threshold <t> --baselines \
--strong-price 0.60 --weak-price 0.03 --out results/mmlu.json
Optional serving:
python scripts/serve_router.py --checkpoint ckpt/mf_repro --text-dim 1024 --port 6060
# client: model="router-mf-<threshold>", base_url=http://localhost:6060/v1
Design decisions & caveats (read before running)
- No Llama-pair training data was generated. The paper's full recipe for a NEW pair regenerates judge battles (~$700 of API judging in the paper). We train on the released preference data (human Arena votes + GPT-4-vs-Mixtral judge battles + MMLU golden battles), which the paper shows transfers across pairs: Sec 5.2 / Table 4 β "we use the same routers as before without any retraining, and only replace the strong and weak model routed to", tested on Llama 3.1 70B/8B.
- Scoring the Llama pair through proxy ids. The 64-name
MODEL_IDStable in the checkpoint has no Llama-3.x entries, so the MF score for Llama-3.3-70B vs 3.1-8B is computed asllama-2-70b-chat(id 30) vsllama-2-7b-chat(id 31) β the closest registered 70B-vs-7B pair. This is an approximation grounded in the paper's transfer result; override with--strong-proxy/--weak-proxy. - MF training code is ours. Upstream ships the router runtime but no MF
training script (lm-sys/RouteLLM issue #65). We implement the objective
BCE(sigmoid-logits of s_a - s_b, y)with ties at 0.5 (--drop-tiesto exclude), using the paper's hyperparameters: Adam, lr 3e-4, weight decay 1e-5, batch 64, ~10 epochs, dim 128. - Embedding consistency. Training embeddings and inference embeddings must
use the same backend. The released
routellm/mf_gpt4_augmentedcheckpoint only works with OpenAItext-embedding-3-small(1536-dim); a freshly trained checkpoint works with the openQwen/Qwen3-Embedding-0.6B(1024-dim) β the default here, no API key needed. - Open-weight scoring. Upstream
MFModel.forward()hardcodes the OpenAI embeddings client;scripts/mf_scorer.pyreproducespred_win_rateexactly from precomputed embeddings instead.
Compute pre-flight (HF Jobs pricing, Oct 2026)
| Step | Flavor | Est. time | Est. cost |
|---|---|---|---|
| prep + train MF + calibrate | cpu-upgrade ($0.03/hr) |
~1 h | ~$0.05 |
| embeddings (166K prompts) | a10g-small ($1.00/hr) |
~1-1.5 h | ~$1.50 |
| smoke eval (64 GSM8K prompts) | strong 1xH200 fp8 ($5/hr) + weak a10g-small ($1/hr) |
~30 min | ~$2.50 |
| full eval (GSM8K + MMLU, 3 policies) | same | ~2-3 h | ~$10-15 |
Serving options for Llama-3.3-70B-Instruct: 1xH200 (141 GB) with vLLM
--quantization fp8 (~70 GB weights), or an AWQ/GPTQ 4-bit build on 1xL40S
48 GB ($1.80/hr). Llama-3.1-8B-Instruct bf16 fits an a10g-small (24 GB,
$1.00/hr). Alternatively use Inference Providers endpoints for either model and
skip local serving entirely.
Expected results
Paper reference points (GPT-4 vs Mixtral): MF router at CPT(50%) reaches MT-Bench 8.8 (95% of GPT-4), MMLU 75 (92% of GPT-4's 81), GSM8K 75 (87% of GPT-4's 86); routers transfer across pairs (Table 4) at ~half the strong-model calls of a random router. For the Llama pair expect router accuracy between always-weak and always-strong, closer to always-strong at the 50%-strong threshold β report what you measure; this repo ships code, not results.
Attribution
- Paper: Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, arXiv:2406.18665
- Code base: lm-sys/RouteLLM (
pip install routellm[serve,eval]==0.2.0) - Data:
lmarena-ai/arena-human-preference-55k,routellm/gpt4_judge_battles,routellm/mmlu_battles(Apache-2.0)