Instructions to use litert-community/Ternary-Bonsai-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/Ternary-Bonsai-8B with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=litert-community/Ternary-Bonsai-8B \ --prompt="Write me a poem"
- Notebooks
- Google Colab
- Kaggle
Ternary-Bonsai-8B β LiteRT-LM (INT2 / ternary, GPU + CPU)
Ternary (INT2) .litertlm bundles of
prism-ml/Ternary-Bonsai-8B-unpacked,
packaged for the LiteRT-LM runtime and
Google AI Edge Gallery.
These run on the GPU backend (recommended β fastest) or the CPU backend. Not the NPU.
An 8B model in the footprint of a 4B. Ternary weights (every weight is β1, 0 or +1, with a scale) bring this Qwen3-8B-class model down to 2.6 GB:
| model | format | download |
|---|---|---|
| Qwen3-8B | Q4_K_M GGUF | 5.03 GB |
| Qwen3-4B | Q4_K_M GGUF | 2.50 GB |
| Ternary-Bonsai-8B (this repo) | INT2 .litertlm |
2.56 GB |
1.7B or 8B?
| Ternary-Bonsai-1.7B | Ternary-Bonsai-8B | |
|---|---|---|
| download | 0.77 GB | 2.56 GB |
| pick it for | speed, older or low-RAM phones, in-app downloads | answer quality, reasoning, longer and harder prompts |
If your phone has the free memory for a ~2.5 GB model, the 8B is the one to try.
Files
| file | ctx | weights | activations | use it when |
|---|---|---|---|---|
bonsai-8b-int2pc-4k-fp16.litertlm |
4096 | INT2 per-channel | fp16 | default β start here |
All bundles carry Bonsai's own chat template (Qwen3 ChatML with the reasoning block intact).
Experimental builds (experimental/)
Longer context or fused attention. Whether these load depends on your app's LiteRT / LiteRT-LM version β many won't. Start with the file above.
| file | ctx | weights | activations | notes |
|---|---|---|---|---|
experimental/bonsai-8b-int2pc-32k-fp16.litertlm |
32768 | INT2 per-channel | fp16 | long context |
experimental/bonsai-8b-int2all-32k-sdpa-fp16.litertlm |
32768 | INT2 (all) | fp16 | fused attention (SDPA), smallest (2.16 GB) |
Ternary-Bonsai-8B's max_position_embeddings is 65536, so the 32k builds do not reach the model's full context.
Performance
Pixel 11 Pro XL (Tensor G6), Google AI Edge Gallery 1.0.23, bonsai-8b-int2pc-4k-fp16.litertlm, 256 prefill + 256
decode tokens, 3 runs, median:
| backend | prefill | decode | time to first token |
|---|---|---|---|
| GPU | 81 tok/s | 5.8 tok/s | 3.3 s |
| CPU | 33 tok/s | 2.3 tok/s | 8.2 s |
The first GPU load takes about 4 minutes while the GPU programs compile. Edge Gallery caches them, so later loads take 15β20 s. The first CPU load takes about 6 s.
Backends
| backend | status |
|---|---|
| GPU | β recommended, fastest |
| CPU | β
works β verified on a Pixel 11 Pro XL (see Performance) and on a desktop x86 CPU via the litert-lm Python package |
| NPU | β not supported by these bundles |
Why CPU works even though XNNPACK has no INT2 kernel: XNNPACK takes the float ops, and the INT2
FULLY_CONNECTED ops fall back to LiteRT's built-in kernels, which handle INT2 directly. Expect CPU to be
noticeably slower than GPU. A YNNPACK-enabled runtime could speed up CPU INT2, but it isn't required.
Sampling defaults
Every bundle ships these in its LlmMetadata, so a LiteRT-LM host picks them up without
any configuration:
| parameter | value |
|---|---|
| type | TOP_P |
| top-k | 20 |
| top-p | 0.85 |
| temperature | 0.5 |
These are the values the bundles were built with. Override them in your host if you want different behaviour.
Usage
Any LiteRT-LM host β the AI Edge Gallery app, or litert_lm_main β with the GPU backend (fastest) or the CPU backend selected.
For sample apps, see google-ai-edge/litert-samples.
Chat template fixed (2026-11-07). Since LiteRT-LM 0.18.0 the runtime hands the chat template message.content as a list of typed parts, and the template embedded in these bundles read only a plain string, so on 0.18+ it rendered an empty user turn and the reply went off topic (found and reported by mlboydaisuke, thank you). Every .litertlm here now carries a template that reads both a plain string and typed parts, for system and user turns, so no --chat-template override is needed and older runtimes keep working. Only the metadata section changed; the weights are byte-for-byte the same. Verified on LiteRT-LM 0.19.0 (1.7B and 8B, CPU backend).
Provenance and licensing
- Base model:
prism-ml/Ternary-Bonsai-8B-unpacked, Apache-2.0 - Built from Qwen3-8B, Copyright 2024 Alibaba Cloud, Apache-2.0
- These bundles are quantized, repackaged derivatives. No weights were retrained.
Created using Bonsai by Prism ML.
Training data: None was used here. These are post-training quantizations and repackagings of the released Bonsai checkpoint; no additional training, fine-tuning, or calibration data was involved. For the base model's training data, see the upstream Prism ML and Qwen3 model cards.
PII: No dataset was collected, processed, or shipped as part of this conversion, so no personally identifiable information is present in these artifacts beyond whatever the upstream released weights already encode.
- Downloads last month
- 526
Model tree for litert-community/Ternary-Bonsai-8B
Base model
prism-ml/Ternary-Bonsai-8B-unpacked