Qwen-3.5-4B-NVFP4A16

Weight-only NVFP4A16 quantization of Qwen/Qwen3.5-4B (revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a), produced with llm-compressor 0.12.0.

This is not a fine-tune โ€” the weights are the official Qwen3.5-4B weights, quantized. It is packaged in the Qwen3_5ForConditionalGeneration layout; the vision tower is bundled unquantized and unused when served with --language-model-only.

Scheme

4-bit NVFP4 weights (E2M1 with an FP8 scale per block of 16), bf16 activations. input_activations is null, so vLLM selects CompressedTensorsW4A16Fp4 โ†’ MarlinNvFp4LinearKernel, not the W4A4 path that most public NVFP4 checkpoints take.

Left in bf16 via ignore: lm_head, embed_tokens, the full visual tower, every linear_attn (GDN) projection, and mtp. Only self_attn and mlp in the full-attention layers are quantized.

GPTQModifier:
  targets: [Linear]
  scheme: NVFP4A16
  block_size: 128
  dampening_frac: 0.01
  actorder: static
  ignore: ['re:.*lm_head', 're:.*embed_tokens', 're:.*visual.*',
           're:.*linear_attn.*', 're:^mtp.*']

Calibration

512 prompts sampled from a 2342-prompt pool of short real-world chat messages wrapped in a few-shot language-identification prompt โ€” not C4 or Pile. Locale mix of the pool: es 1360, en 471, und 131, ar 93, de 68, uk 47, ru 36, and a long tail. Length buckets skew short (860 samples at 11-20 chars, 541 at 4-10). Sequence length 512.

Evaluation

Language identification, top-1 accuracy, greedy, scores read from first-token logprobs. Compared against the same base model served with vLLM's online per-tensor FP8 forced into Marlin W8A16 (VLLM_TEST_FORCE_FP8_MARLIN=1).

checkpoint uk/ru bench (n=347) hard-cases bench (n=1309)
this (NVFP4A16) 330 (95.10%) 1146 (87.55%)
base, online FP8 W8A16 333 (95.97%) 1144 (87.39%)
int8 W8A16 group-32 329 (94.81%) 1144 (87.39%)
int4 W4A16 group-128 331 (95.39%) 1135 (86.71%)
NVFP4 W4A4 326 (93.95%) 1132 (86.48%)
bf16, no quantization 329 (94.81%) โ€”

No pair differs significantly (McNemar exact, all p > 0.4; vs the FP8 baseline p=0.45 and p=0.86). Notably bf16 does not top the table โ€” on this task the spread across schemes is measurement noise on the ukโ†”ru boundary, so treat the ordering above as ties, not a ranking.

Both benches are disjoint from the calibration pool.

Serving

vllm serve CodeStreet/Qwen-3.5-4B-NVFP4A16 \
  --language-model-only --max-model-len 512 --max-logprobs 128 \
  --gpu-memory-utilization 0.12 --trust-remote-code

gpu-memory-utilization needs more headroom than a fully-quantized NVFP4 checkpoint: the conservative ignore list keeps ~3.3 GB in bf16 (GDN projections 2.0 GB, embed_tokens 1.3 GB). At 0.07 on a 96 GB card the engine fails with max_num_seqs exceeds available Mamba cache blocks. Requires Blackwell (sm_120) or newer for the NVFP4 kernels.

Downloads last month
17
Safetensors
Model size
5B params
Tensor type
BF16
ยท
U8
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for CodeStreet/Qwen-3.5-4B-NVFP4A16

Finetuned
Qwen/Qwen3.5-4B
Quantized
(689)
this model