Ternary-Bonsai-8B β€” LiteRT-LM (INT2 / ternary, GPU + CPU)

Ternary (INT2) .litertlm bundles of prism-ml/Ternary-Bonsai-8B-unpacked, packaged for the LiteRT-LM runtime and Google AI Edge Gallery.

These run on the GPU backend (recommended β€” fastest) or the CPU backend. Not the NPU.

An 8B model in the footprint of a 4B. Ternary weights (every weight is βˆ’1, 0 or +1, with a scale) bring this Qwen3-8B-class model down to 2.6 GB:

model format download
Qwen3-8B Q4_K_M GGUF 5.03 GB
Qwen3-4B Q4_K_M GGUF 2.50 GB
Ternary-Bonsai-8B (this repo) INT2 .litertlm 2.56 GB

1.7B or 8B?

Ternary-Bonsai-1.7B Ternary-Bonsai-8B
download 0.77 GB 2.56 GB
pick it for speed, older or low-RAM phones, in-app downloads answer quality, reasoning, longer and harder prompts

If your phone has the free memory for a ~2.5 GB model, the 8B is the one to try.

Files

file ctx weights activations use it when
bonsai-8b-int2pc-4k-fp16.litertlm 4096 INT2 per-channel fp16 default β€” start here

All bundles carry Bonsai's own chat template (Qwen3 ChatML with the reasoning block intact).

Experimental builds (experimental/)

Longer context or fused attention. Whether these load depends on your app's LiteRT / LiteRT-LM version β€” many won't. Start with the file above.

file ctx weights activations notes
experimental/bonsai-8b-int2pc-32k-fp16.litertlm 32768 INT2 per-channel fp16 long context
experimental/bonsai-8b-int2all-32k-sdpa-fp16.litertlm 32768 INT2 (all) fp16 fused attention (SDPA), smallest (2.16 GB)

Ternary-Bonsai-8B's max_position_embeddings is 65536, so the 32k builds do not reach the model's full context.

Performance

Pixel 11 Pro XL (Tensor G6), Google AI Edge Gallery 1.0.23, bonsai-8b-int2pc-4k-fp16.litertlm, 256 prefill + 256 decode tokens, 3 runs, median:

backend prefill decode time to first token
GPU 81 tok/s 5.8 tok/s 3.3 s
CPU 33 tok/s 2.3 tok/s 8.2 s

The first GPU load takes about 4 minutes while the GPU programs compile. Edge Gallery caches them, so later loads take 15–20 s. The first CPU load takes about 6 s.

Backends

backend status
GPU βœ… recommended, fastest
CPU βœ… works β€” verified on a Pixel 11 Pro XL (see Performance) and on a desktop x86 CPU via the litert-lm Python package
NPU ❌ not supported by these bundles

Why CPU works even though XNNPACK has no INT2 kernel: XNNPACK takes the float ops, and the INT2 FULLY_CONNECTED ops fall back to LiteRT's built-in kernels, which handle INT2 directly. Expect CPU to be noticeably slower than GPU. A YNNPACK-enabled runtime could speed up CPU INT2, but it isn't required.

Sampling defaults

Every bundle ships these in its LlmMetadata, so a LiteRT-LM host picks them up without any configuration:

parameter value
type TOP_P
top-k 20
top-p 0.85
temperature 0.5

These are the values the bundles were built with. Override them in your host if you want different behaviour.

Usage

Any LiteRT-LM host β€” the AI Edge Gallery app, or litert_lm_main β€” with the GPU backend (fastest) or the CPU backend selected.

For sample apps, see google-ai-edge/litert-samples.

Chat template fixed (2026-11-07). Since LiteRT-LM 0.18.0 the runtime hands the chat template message.content as a list of typed parts, and the template embedded in these bundles read only a plain string, so on 0.18+ it rendered an empty user turn and the reply went off topic (found and reported by mlboydaisuke, thank you). Every .litertlm here now carries a template that reads both a plain string and typed parts, for system and user turns, so no --chat-template override is needed and older runtimes keep working. Only the metadata section changed; the weights are byte-for-byte the same. Verified on LiteRT-LM 0.19.0 (1.7B and 8B, CPU backend).

Provenance and licensing

Created using Bonsai by Prism ML.

Training data: None was used here. These are post-training quantizations and repackagings of the released Bonsai checkpoint; no additional training, fine-tuning, or calibration data was involved. For the base model's training data, see the upstream Prism ML and Qwen3 model cards.

PII: No dataset was collected, processed, or shipped as part of this conversion, so no personally identifiable information is present in these artifacts beyond whatever the upstream released weights already encode.

Downloads last month
526
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for litert-community/Ternary-Bonsai-8B

Quantized
(23)
this model