Qwen3-4B-Runner-ToolUse-Q4_K_M

This LoRA was trained directly through the quantized GGUF that serves it. No FP16 training copy. No Python training framework. The trainer's forward pass is the inference forward pass — Xyntetik Runner (a single-binary C11 inference engine) trained this adapter against the frozen Q4_K_M inference artifact on CPU, and the result is byte-reproducible: run the same command on the same inputs and you get this exact file, sha for sha.

TL;DR

	json parses	right tool	schema args	exact call
Qwen3-4B Q4_K_M, base	1.000	0.724	0.724	0.690
+ this adapter	1.000	1.000	1.000	1.000

Held-out tool-calling eval (29 prompts, greedy, temperature 0): pick the right tool among four schema-distinct tools, emit schema-shaped JSON args, refuse requests no available tool fits, no prose around the JSON.

Reproducibility record (measured, not asserted)

Everything below is from the machine-written provenance record (Qwen3-4B-ToolUse-LoRA.gguf.train.json, committed beside the adapter):

	sha256
base (Qwen_Qwen3-4B-Q4_K_M.gguf, bartowski's conversion)	fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3
dataset (data/train.jsonl, deterministic generator)	4c3feca6afa9d776b0e5d08afb6ac4b134bee93a94548dd934a74b2c944d6a5a
this adapter	ea38f80c33d381c9aa62c874920d2fa058dd307e7186e9b7c347d782f35b88e4

Config: rank 8 on every projection (attention q/k/v/output + FFN gate/up/down), alpha 16, AdamW (0.9/0.999, wd 0.01), lr 1e-4, 316 steps (2 epochs of 158 examples), seed 0, ctx 128. Loss 0.676 → 0.000006.

Determinism was verified on this exact base before training the release adapter: two independent 40-step runs with identical inputs produced byte-identical adapter files (35a19a2435fd717875929a098019250c7b7b5c3e1b669abec87e20b3fa1916f1, both). Runner gates this property in its test suite (same data + same seed → byte-identical adapter); the engine design notes are in docs/adaptation-engine.md.

Use it

# serve the base with the adapter (CPU)
runner -m Qwen_Qwen3-4B-Q4_K_M.gguf --lora Qwen3-4B-ToolUse-LoRA.gguf --serve --gpu off

# or reproduce the adapter from scratch — you should get the same sha256
python3 scripts/make-tooluse-data.py data/
runner -m Qwen_Qwen3-4B-Q4_K_M.gguf --train data/train.jsonl \
    --train-steps 316 --lr 1e-4 --train-out adapter.gguf

Runner ≥ v0.2.0 (releases); this adapter was produced at commit 3f7cd50 (the v0.2.0 training code). The adapter GGUF uses the llama.cpp adapter naming convention.

The three-precision study (study/)

The same task, same seed, same data, same config, trained through three precisions of the same base — does the quantization you train through change what you learn?

trained through	eval on own base	eval on Q4_K_M base	adapter sha256 (first 16)
bf16	1.000 exact	1.000 exact	24d5f02122bc7436
Q8_0	1.000 exact	1.000 exact	a2c70991d7421bab
Q4_K_M	1.000 exact	—	eda3c58491f9c654

Weight-space divergence between the learned adapters (all-parameter cosine / relative L2):

pair	cosine	rel L2
bf16 vs Q8_0	0.99983	0.019
bf16 vs Q4_K_M	0.99259	0.122
Q8_0 vs Q4_K_M	0.99276	0.120

Read: training through Q8_0 learns essentially the same adapter as training through bf16 (2% relative difference). Training through the 4-bit base learns a measurably different adapter (12% relative divergence) — so "fine-tune FP16, then quantize for deployment" and "adapt the deployed quant directly" are not the same operation — yet on this task's supervised, wide-margin decisions the difference costs nothing: every adapter scores 1.000, and the bf16- and Q8-trained adapters drop onto the Q4_K_M serving base at 1.000 too. Read that scope literally: it is not free everywhere, and the two refinements below show where the cost actually shows up. All three study adapters and their provenance records are in study/.

Follow-up: the divergence is behavioral (evals/logprob-study/). Prompted by an independent reproduction of this repo's determinism claims (HF forum — which also verified adapter byte-equality on a Tesla T4 and correctly proposed logprobs as the next instrument): all 29 held-out prompts with gold completions, scored through the same Q4_K_M serving base under each study adapter, 3,195 positions.

pair	weight rel L2	mean |Δlogprob|	positions >1 nat
bf16-ad vs Q8-ad	0.019	0.020	0%
bf16-ad vs Q4-ad	0.122	0.146	2.2%
base vs any adapter	—	0.32–0.40	6–8%

The Q4-trained adapter differs from the bf16-trained one by ~40% of the entire adapter effect in logprob space; the 6x weight-space gap maps to a 7.4x logprob gap.

Refinement (2026-08-25, from the same independent reviewer who reproduced the determinism claims): splitting those positions by prompt vs gold completion changes what the aggregate measures. On the supervised completion region the three adapters nearly coincide (bf16-vs-Q4 mean |dlogprob| ~0.0003 nat, no position near 1 nat, verified from the published files); almost all of the 0.146 aggregate lives in the UNSUPERVISED prompt region. So the honest reading is: the precision path leaves a real function-level fingerprint, largely outside what the completion loss supervised, while the supervised decisions stay wide-margin. Those are two different observables. The same reviewer then supplied the missing piece: on deliberately ambiguous requests the fingerprint DOES cross a decision boundary. On "tell me what README.md says and translate it", the bf16- and Q8-trained adapters call read_file (margins ~0.49 nat) while the Q4-trained adapter confidently answers none (margin 2.17 nat), and the split survives full deterministic JSON generation. One constructed case, an existence proof rather than a prevalence estimate, and exactly the "where does weight divergence become behavioral" answer this study was reaching for. A small decision-boundary lane (top1/top2 margins over ambiguous tool choices) is the planned follow-up.

Systematized (2026-08-30, same reviewer). That constructed case is now a screened one. He built a 36-prompt tool-choice boundary bank across five ambiguity families (list_dir/search_files, read_file/search, read_file/write_file multi-intent, config browsing, and available-tool/none), kept the cases whose top1-top2 legal-choice margin sits at or under 3 nat, and ran the base plus all three study adapters on the survivors, generating the full JSON deterministically wherever the branches disagreed. Seven cases passed the screen, and the split above is the one that reproduced. Three constraints travel with it, all his, and they bound what this result may be used for:

• The screen selects on the Q4 adapter's own margin, so the seven selected cases cannot estimate any general margin distribution. One split out of seven out of 36 is an existence proof, not a rate.
• There is no monotonic precision law here. Several cases had smaller Q4 margins, not all did, and one case had Q8 as the tightest condition. What this supports is that some decision boundaries are far more sensitive to the training precision path than others, non-monotonically in bit width. It does not support "lower training precision degrades tool choice", which his own data refutes.
• The bank carries no gold labels, deliberately: labeling would convert a decision-sensitivity probe into a quality benchmark and quietly change the question.

Notebooks, clean and executed with reference outputs: John6666/forum3.

The merge study (merged/) — or: why this repo ships an adapter

Runner's --merge-lora folds the adapter into the base weights (W' = W + (alpha/r)·B·A, deterministic: same inputs → byte-identical merged file) so the result runs in any GGUF runtime. We merged this adapter into three output precisions of the same Q4_K_M base and re-ran the held-out eval on each standalone file:

merged into	right tool	exact call	verdict
Q4_K_M (base's own type)	0.724	0.690	adapter erased — scores exactly the base
Q8_0	1.000	1.000	behavior fully preserved
F16	1.000	1.000	behavior fully preserved

The 4-bit merge is not slightly worse — it reproduces the base's numbers to the prompt. Byte-level comparison shows why: after requantizing to Q4_K_M, only 1.45% of weight bytes differ from the base — the LoRA delta is small relative to the 4-bit grid step, so quantization rounds it back to the original codes almost everywhere. The visible symptom (via stock llama.cpp b10581, raw completion): the Q8_0 and F16 merges answer with JSON only, exactly as trained; the Q4_K_M merge answers, then drifts into conversational prose — the base's behavior.

Deployment guidance, measured: to serve 4-bit with this adaptation, serve base + --lora (exact, and how the 1.000 row above is produced). To ship a standalone merged file, merge into Q8_0 or wider and measure it — this repo's merged/merged-Q8_0.gguf is that artifact, verified 1.000 on the eval and runnable in stock llama.cpp. Merging into a 4-bit base is where fine-tunes silently disappear; if you do it anywhere, eval the merged file, not the adapter.

The survival threshold, measured (follow-up sweep): the same adapter merged at five --lora-scale values into the same Q4_K_M base, every merged file evaled standalone, served base+adapter as the control:

scale	merged-Q4 exact call	served exact call	bytes changed
0.5	0.690 (base)	1.000	1.15%
1	0.690 (base)	1.000	1.45%
2	0.690 (base)	1.000	2.05%
4	0.828	1.000	2.87%
8	1.000	0.138	4.60%

Survival is monotone in delta magnitude, as the rounding mechanism predicts — and note the inversion at 8×: the exact over-scaled adapter breaks the served model, while the 4-bit grid filters the same weights back to 1.000. Quantization acts as a filter on your fine-tune with a pass-band you don't control. The sweep evals are in evals/sweep-*.json.

Interop, both directions: this adapter scores the same 1.000/1.000/1.000/1.000 on the full held-out eval when served by stock llama.cpp (b10581, llama-server --lora, raw completion) — the eval output is evals/llamacpp-served-adapter.json. And as of runner main, --lora accepts the F16 adapters llama.cpp's convert_lora_to_gguf emits (found by loading a community adapter, which the old F32-only check refused).

The Q4_K_M and F16 merges are not uploaded (one is behaviorally the base, the other is 8 GB); their provenance records with sha256s are in merged/, and because the merge is deterministic, one command reproduces either file byte-for-byte:

runner -m Qwen_Qwen3-4B-Q4_K_M.gguf --lora Qwen3-4B-ToolUse-LoRA.gguf \
    --merge-lora merged-Q4_K_M.gguf            # sha256 17443f0d55f5bb33…

(--merge-lora needs runner newer than v0.2.0 — main at 54fe720+.)

Honest limits

• The task is narrow and synthetic (four tools, template-generated requests, 29 held-out prompts). This demonstrates the training mechanism and its reproducibility, not general tool-use capability.
• 316 steps at lr 1e-4 on 158 examples is memorization-grade training — appropriate for a format task, not a recipe for knowledge tasks.
• The precision-study equivalence is measured on this task; harder tasks may separate the adapters. That experiment is now a one-command rerun for anyone, which is rather the point.
• Training runs on CPU (this adapter: ~45 s/step at 4B on a 128-thread EPYC-class host, with runner v0.2.0). Runner main has since position-batched and threaded the backward under a byte-exact contract: the same config now trains at ~20 s/step, and the adapter bytes are gated identical across the old binary, the new one, any thread count, and the optional CUDA assist — so this artifact's reproduce-the-sha command works unchanged on current builds, just 2.3× faster.

Files

• Qwen3-4B-ToolUse-LoRA.gguf — the adapter (+ .train.json provenance)
• data/ — the deterministic dataset (train + held-out eval)
• evals/ — every eval JSON quoted above, unedited (incl. the three merged-artifact evals)
• study/ — the three precision-study adapters, provenance, divergence
• merged/ — the standalone merged Q8_0 GGUF (runs in any GGUF runtime, verified 1.000 on the eval and smoke-tested in llama.cpp b10581) + .merge.json provenance for all three merge targets