FreeToken community model releases by qualitymaterial

Qwen3.8-35B-A3B-Distill-NVFP4-FreeToken

A community mixed-precision conversion for local inference with FreeToken, tested on an RTX 5070 Ti 16 GB and 64 GB system RAM.

This release converts Empero's Qwen3.8-35B-A3B-Distill to NVFP4/FP8 while retaining selected components in BF16. It includes the converted weights, a portable launcher, exact runtime settings, reproduction instructions, checksums, and evaluation records.

The goal was practical: use a 35B MoE model on the workstation I already own, keep a large reusable prefix cache, and load or unload the model through FreeToken Desktop. Expert weights live in system RAM with a GPU expert cache. The approximately 24.19 GB checkpoint does not fit entirely in 16 GB VRAM.

At a glance Release details
Model Approximately 35B total parameters, 3B active per token
Quantization NVFP4 experts, selected FP8 attention projections, retained BF16 components
Tested workstation RTX 5070 Ti 16 GB, Ryzen 9 9900X, 64 GB RAM, Windows
Tested runtime FreeToken Desktop v0.2.0-beta.23; engine 0.1.3+gc8ed699cb
Deployment Expert weights offloaded to RAM; raw safetensors loading
Native context limit 262,144 tokens per sequence
Allocated attention cache 347,136 shared token slots, displayed as 339K
Long prompt tested 138,187 input tokens
License Apache-2.0, following the source declaration

This is an independent community conversion. It is not an official Alibaba or NVIDIA release. The recorded tests establish that this checkpoint ran with the documented setup; they do not establish a quality advantage over the original Qwen3.6 model or the BF16 source.

Quick start: FreeToken Desktop

The steps below were tested with FreeToken Desktop v0.2.0-beta.23 on Windows. Other runtimes and GPUs have not been validated by this release.

  1. Download this repository to a permanent local folder. For example, with the Hugging Face CLI:

    hf download qualitymaterial/Qwen3.8-35B-A3B-Distill-NVFP4-FreeToken --local-dir 'C:\Models\Qwen3.8-35B-A3B-Distill-NVFP4-FreeToken'
    
  2. In FreeToken Desktop, open Models and select the local model-folder import control.

  3. Select the downloaded folder containing config.json and the safetensors files, then start the imported model.

  4. In Console, enter the settings below and select Apply.

    Setting Tested value
    KV cache 347,136 tokens
    Expert cache 512 slots
    Mamba state slots 24
  5. FreeToken displays the attention cache as 339K and saves the applied settings for the model's next launch. Use Models → Start/Stop to load or unload it.

Reduce the cache settings if your available memory differs. Desktop applications also consume memory, so the same nominal hardware specifications do not guarantee the same available allocation.

339K is a shared cache pool, not a validated 339K conversation window. The native per-sequence limit is 262,144 tokens. The longest prompt tested here was 138,187 tokens; context extension beyond the native limit and a full 262K prompt were not tested.

The default local API is http://127.0.0.1:1919/v1. Query /v1/models for the runtime's served model ID. No FTW preconversion was required for the recorded tests.

Portable launcher

Start-FreeToken-Distill.ps1 requires an installed, running FreeToken daemon. Run it from the downloaded repository folder; it defaults to that folder or accepts an explicit model path:

.\Start-FreeToken-Distill.ps1 -ModelPath 'C:\Models\Qwen3.8-35B-A3B-Distill-NVFP4-FreeToken'
.\Start-FreeToken-Distill.ps1 -Action Stop

Start can switch away from the daemon's current model. Stop stops its current model. Desktop import is still required for a library entry, and Console Apply saves settings for subsequent desktop launches.

The packaged launcher is adapted from the tested local launcher. Its parameterized wrapper was syntax-checked, but was not separately exercised in a new model restart.

freetoken-settings.json records the exact runtime settings:

{"num_pages": 347136, "moe_cache_size": 512, "num_mamba_slots": 24}

This is the request body for POST /v1/cache/rebuild on the inference API. Keep it separate from config.json, which defines the model architecture and quantization.

Recorded results

These are the initial recorded tests on the workstation above, not a new benchmark run. Measurements and derived statistics are in evaluation/release-stats.json.

Test Observed result
Short response Passed
Automatic function call with JSON arguments Passed smoke check
Tool-result round trip Passed smoke check
Reasoning-enabled arithmetic Correct on the tested example
Short prefix reuse 3,648 cached tokens reported
Synthetic long prompt 138,187 input tokens; beginning, middle, and end markers retrieved correctly
First long request 107.0 seconds end to end
Identical-prefix repeat 2.57 seconds end to end; 138,176 cached tokens reported
Short coding decode Approximately 53–62 tokens/s in steady server log intervals
Desktop controls Stop unloaded; Start reloaded successfully

The repeated long request reused 99.992% of the input tokens and completed 41.63× faster end to end in this synthetic test. That ratio primarily reflects avoiding repeated prefix processing. It is not a general decode-speed improvement.

The coding speed is a log-based observation for a short response, not an aggregate benchmark. That response also included Markdown fences despite an instruction requesting bare code. Successful basic tool calls do not establish reliable multi-step agent performance.

Cache allocation

Metric Value Evidence
Attention cache pool 347,136 token slots, displayed as 339K Engine cache geometry
Attention cache bytes per token 20,480 bytes = 20 KiB Engine cache geometry
Attention cache allocation 6.621 GiB Calculated from geometry
GPU expert cache 512 slots across 40 MoE layers Engine cache geometry
Expert cache allocation 0.847 GiB Calculated from geometry
Recurrent state slots 24 Engine cache geometry
Recurrent state allocation 1.439 GiB Calculated from geometry
Combined configured caches 8.907 GiB Sum of the three allocations; excludes weights and runtime buffers
Long-prompt cached fraction on repeat 99.992% 138,176 / 138,187 input tokens
End-to-end repeat speedup 41.63× 107.0166 / 2.57092 seconds

No maximum stable concurrency test was performed. The 24 recurrent-state slots do not imply that 24 simultaneous long conversations fit.

Evaluation artifacts

Scripts and synthetic-response records are in evaluation/. synthetic-long-prompt.json contains the exact request body reconstructed deterministically from the original test script: 8,192 reference rows, marker substitutions, messages, and generation settings.

This request file was reconstructed after the test rather than captured as a request log. The server reported 138,187 prompt tokens for that request; the JSON file's size is not its token count. synthetic-long-prompt.sha256 verifies the reconstructed request file.

No broad benchmark or matched real-world DeepSeek Harness comparison has been completed. Quantization accuracy relative to the BF16 source has not been measured.

Why I built this

I wanted a local model I could use for sustained agent work in DeepSeek Harness Desktop on the hardware I already own. In my local testing, FreeToken gave me the most useful combination of performance and cache capacity. That is my experience with this setup, not a controlled comparison against every inference engine.

My working Qwen3.6 setup already had a 339K cache pool. I wanted to try the newer community distillation without giving up that allocation or buying a larger GPU. I also wanted a permanent checkpoint I could load, unload, and share with enough detail for someone else to reproduce the setup.

With MoE experts in RAM, the whole checkpoint does not have to fit in VRAM. GPU memory still has to cover resident tensors, working buffers, the expert cache, recurrent state, and the attention cache. Increasing one cache leaves less room for the others. This conversion let me keep a relatively small GPU expert cache and reserve more VRAM for reusable context. The largest possible cache is not necessarily the fastest configuration.

RunPod supplied temporary conversion hardware. Daily inference and all tests reported here ran locally; continued GPU rental is not required to use this checkpoint with the documented local setup.

Conversion and reproduction

The architecture is inherited from Qwen3.6-35B-A3B: approximately 35B total parameters, 3B active per token, and hybrid linear/full attention. Empero performed the distillation. This release changes weight precision, not training or architecture.

Component Stored precision
Routed and shared experts NVFP4, weight-only W4A16
Selected full/linear attention projections FP8, with dynamic activation quantization configuration
Embeddings, output head, routers, norms, sensitive recurrent projections, convolution, vision, and MTP Retained BF16 per recipe

The conversion uses data-free post-training quantization. It does not use calibration-based quantization or quantization-aware training.

Conversion detail Value
Source empero-ai/Qwen3.8-35B-A3B-Distill
Source revision bcc2dbe2f21b213625df2dc1a5a690212373af07
Recipe Qwen DGX Spark conversion project
Conversion dependency llmcompressor==0.14.0
Conversion GPU RTX PRO 4500 Blackwell, via RunPod
Output 21 main safetensors shards plus an MTP shard
Checkpoint payload 24,193,559,244 bytes, approximately 24.19 GB / 22.53 GiB, including tokenizer/configuration files

See REPRODUCE.md for the pinned recipe reference and conversion steps. The full transitive dependency environment was not captured, so byte-identical reproduction is not guaranteed. SHA256SUMS verifies the original 32 checkpoint payload files; release documentation is outside that manifest.

The recipe repository did not declare a license when checked on 2026-10-02. Its script is linked rather than redistributed here. Contact the recipe author or the DGX-Spark project if you plan to redistribute the recipe itself.

Workstation memory

Specification Verified value
GPU NVIDIA GeForce RTX 5070 Ti
GPU memory reported by driver 16,303 MiB, approximately 15.92 GiB
CPU AMD Ryzen 9 9900X, 12 cores / 24 logical processors
System RAM 64 GB installed; FreeToken reports approximately 61.6 GiB usable
Operating system Windows

The payload alone exceeds GPU memory. This deployment uses system RAM and the GPU together. Available memory, background applications, and cache settings affect whether another machine can reproduce the same allocation.

Limitations

  • Long-context quality: Empero warns that its student was trained on 8,192-token examples and that long-form generation and long-context behavior may degrade relative to the base. The synthetic marker-retrieval test does not remove that limitation.
  • Agent reliability: Tool-call and tool-result smoke checks do not establish reliable multi-step agent behavior.
  • Quality comparison: Quantization accuracy against the BF16 source and a broad benchmark have not been measured.
  • Context and concurrency: The 339K display describes a shared cache pool. A full 262K prompt, context extension beyond the native limit, and maximum stable concurrency were not tested.
  • Vision and video: These behaviors were not evaluated; Empero's distillation was text-only. Vision tensors are retained.
  • MTP: MTP tensors are retained, but speculative decoding was not tested.
  • Checkpoint size: The output head, vision, MTP, and other retained BF16 tensors contribute to the payload size.
  • Compatibility: Validation covers the documented FreeToken version and workstation. The conversion recipe differs from NVIDIA's original Qwen3.6 quantization; this is not an NVIDIA-certified artifact.

Attribution and license

Weights follow the source's Apache-2.0 declaration. See LICENSE and NOTICE.md.

Downloads last month
678
Safetensors
Model size
20B params
Tensor type
BF16
·
F8_E4M3
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for qualitymaterial/Qwen3.8-35B-A3B-Distill-NVFP4-FreeToken

Quantized
(23)
this model