RouteLLM reproduction β€” Llama-3.3-70B vs Llama-3.1-8B

A runnable reproduction of RouteLLM: Learning to Route LLMs with Preference Data (arXiv:2406.18665, Ong et al., ICLR 2025), built on the official lm-sys/RouteLLM implementation. A lightweight matrix-factorization router learns P(strong model wins | query) from pairwise preference battles, and routes each query to meta-llama/Llama-3.3-70B-Instruct (strong) or meta-llama/Llama-3.1-8B-Instruct (weak) to cut cost without proportional quality loss. The paper reports ~2-3.7x cost savings at matched quality with this method (for GPT-4 vs Mixtral).

Verified assets

Role HF id Notes
D_arena (human battles) lmarena-ai/arena-human-preference-55k 57.5K rows, 9 cols
D_judge (GPT-4-judge battles) routellm/gpt4_judge_battles 109.1K rows
D_gold (MMLU golden battles) routellm/mmlu_battles 1.5K rows, MMLU validation split
Released MF checkpoint routellm/mf_gpt4_augmented 205K params, GPT-4/Mixtral pair
Threshold scores (upstream) routellm/lmsys-arena-human-preference-55k-thresholds per-router scores, 57.5K rows
Strong model meta-llama/Llama-3.3-70B-Instruct gated β€” accept the license first
Weak model meta-llama/Llama-3.1-8B-Instruct gated β€” accept the license first
Embeddings (open) Qwen/Qwen3-Embedding-0.6B 1024-dim, Apache-2.0
Benchmarks openai/gsm8k (main/test, 1319), cais/mmlu (all/validation, 1531) 8-shot / 5-shot as in the paper

All three battle datasets share the schema id, model_a, model_b, prompt, response_a, response_b, winner_model_a, winner_model_b, winner_tie; prompt is a JSON list of turns and routers are trained on the first turn (the same convention as upstream routellm/calibrate_threshold.py).

Pipeline

prep_battles.py   -> data/battles_{arena,judge,mmlu,all}.parquet  (prompt, model_a, model_b, y)
embed_prompts.py  -> data/embeddings_*.npy                        (Qwen3-0.6B, open, or OpenAI)
train_mf.py       -> ckpt/mf_repro/                               (~205K params, minutes on CPU)
calibrate.py      -> ckpt/mf_repro/thresholds.json                (quantile of scores)
eval_routed.py    -> results/eval.json                            (accuracy / %strong / cost)
serve_router.py   -> OpenAI-compatible routing proxy (optional)
pip install -r requirements.txt          # pins routellm==0.2.0
python scripts/prep_battles.py

# Embeddings β€” the SAME backend must be used for training and all scoring.
python scripts/embed_prompts.py --battles data/battles_all.parquet \
    --backend qwen3 --out data/embeddings_all.npy
python scripts/embed_prompts.py --battles data/battles_arena.parquet \
    --backend qwen3 --out data/embeddings_arena.npy

python scripts/train_mf.py --battles data/battles_all.parquet \
    --embeddings data/embeddings_all.npy --out-dir ckpt/mf_repro \
    --push-to-hub <your-namespace>/mf_llama_repro

python scripts/calibrate.py --embeddings data/embeddings_arena.npy \
    --checkpoint ckpt/mf_repro --text-dim 1024 --strong-pcts 0.5 0.25 0.8
# e.g. prints: For 50% strong-model calls, threshold = 0.xxxxx

Evaluation (two OpenAI-compatible endpoints, e.g. vLLM):

export STRONG_BASE_URL=http://localhost:8000/v1  STRONG_MODEL=meta-llama/Llama-3.3-70B-Instruct
export WEAK_BASE_URL=http://localhost:8001/v1    WEAK_MODEL=meta-llama/Llama-3.1-8B-Instruct

# smoke: 64 prompts, router policy only
python scripts/eval_routed.py --benchmark gsm8k --limit 64 --threshold <t> --out results/smoke.json
# full: router + always-strong + always-weak baselines
python scripts/eval_routed.py --benchmark gsm8k --threshold <t> --baselines \
    --strong-price 0.60 --weak-price 0.03 --out results/gsm8k.json
python scripts/eval_routed.py --benchmark mmlu  --threshold <t> --baselines \
    --strong-price 0.60 --weak-price 0.03 --out results/mmlu.json

Optional serving:

python scripts/serve_router.py --checkpoint ckpt/mf_repro --text-dim 1024 --port 6060
# client: model="router-mf-<threshold>", base_url=http://localhost:6060/v1

Design decisions & caveats (read before running)

  1. No Llama-pair training data was generated. The paper's full recipe for a NEW pair regenerates judge battles (~$700 of API judging in the paper). We train on the released preference data (human Arena votes + GPT-4-vs-Mixtral judge battles + MMLU golden battles), which the paper shows transfers across pairs: Sec 5.2 / Table 4 β€” "we use the same routers as before without any retraining, and only replace the strong and weak model routed to", tested on Llama 3.1 70B/8B.
  2. Scoring the Llama pair through proxy ids. The 64-name MODEL_IDS table in the checkpoint has no Llama-3.x entries, so the MF score for Llama-3.3-70B vs 3.1-8B is computed as llama-2-70b-chat (id 30) vs llama-2-7b-chat (id 31) β€” the closest registered 70B-vs-7B pair. This is an approximation grounded in the paper's transfer result; override with --strong-proxy/--weak-proxy.
  3. MF training code is ours. Upstream ships the router runtime but no MF training script (lm-sys/RouteLLM issue #65). We implement the objective BCE(sigmoid-logits of s_a - s_b, y) with ties at 0.5 (--drop-ties to exclude), using the paper's hyperparameters: Adam, lr 3e-4, weight decay 1e-5, batch 64, ~10 epochs, dim 128.
  4. Embedding consistency. Training embeddings and inference embeddings must use the same backend. The released routellm/mf_gpt4_augmented checkpoint only works with OpenAI text-embedding-3-small (1536-dim); a freshly trained checkpoint works with the open Qwen/Qwen3-Embedding-0.6B (1024-dim) β€” the default here, no API key needed.
  5. Open-weight scoring. Upstream MFModel.forward() hardcodes the OpenAI embeddings client; scripts/mf_scorer.py reproduces pred_win_rate exactly from precomputed embeddings instead.

Compute pre-flight (HF Jobs pricing, Oct 2026)

Step Flavor Est. time Est. cost
prep + train MF + calibrate cpu-upgrade ($0.03/hr) ~1 h ~$0.05
embeddings (166K prompts) a10g-small ($1.00/hr) ~1-1.5 h ~$1.50
smoke eval (64 GSM8K prompts) strong 1xH200 fp8 ($5/hr) + weak a10g-small ($1/hr) ~30 min ~$2.50
full eval (GSM8K + MMLU, 3 policies) same ~2-3 h ~$10-15

Serving options for Llama-3.3-70B-Instruct: 1xH200 (141 GB) with vLLM --quantization fp8 (~70 GB weights), or an AWQ/GPTQ 4-bit build on 1xL40S 48 GB ($1.80/hr). Llama-3.1-8B-Instruct bf16 fits an a10g-small (24 GB, $1.00/hr). Alternatively use Inference Providers endpoints for either model and skip local serving entirely.

Expected results

Paper reference points (GPT-4 vs Mixtral): MF router at CPT(50%) reaches MT-Bench 8.8 (95% of GPT-4), MMLU 75 (92% of GPT-4's 81), GSM8K 75 (87% of GPT-4's 86); routers transfer across pairs (Table 4) at ~half the strong-model calls of a random router. For the Llama pair expect router accuracy between always-weak and always-strong, closer to always-strong at the 50%-strong threshold β€” report what you measure; this repo ships code, not results.

Attribution

  • Paper: Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, arXiv:2406.18665
  • Code base: lm-sys/RouteLLM (pip install routellm[serve,eval]==0.2.0)
  • Data: lmarena-ai/arena-human-preference-55k, routellm/gpt4_judge_battles, routellm/mmlu_battles (Apache-2.0)
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Paper for subhash4face/routellm-llama-repro