VibeVoice-ASR-DFlash2-Drafter

A DFlash 2 block drafter for VibeVoice-ASR (7B) (Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM), for speculative decoding in vibevoice.c. It proposes 8 tokens in one pass; vibevoice.c checks them in one pass of the model and keeps the ones the model agrees with, plus one of its own.

The check is exact: every checked row is computed with the arithmetic of the model's own one-token decode step, so the transcript with this drafter is byte-for-byte the transcript without it β€” words, timestamps, speakers. The drafter only changes how many passes it takes.

One thing changes with --draft: vibevoice.c then attends with flashinfer (--attn auto picks it), because flashinfer checks several rows in one pass over the cache where the default fa2 walks it once per row. The transcript is then byte-for-byte that of --attn flashinfer without a drafter β€” on this model the default's words, with a timestamp 10 ms apart on a 32-minute file. --attn fa2 --draft keeps the default's transcript bit for bit, and gains less on long recordings.

Use

Needs vibevoice.c with DFlash 2 support: branch dflash2 (PR #48), in the next release. The model and this drafter in one download: Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM-DFlash2 (the INT4 drafter in drafter/, used without --draft).

vv_cli --model ./VibeVoice-ASR-AWQ-W4A16-ASYM --audio talk.wav --draft ./VibeVoice-ASR-DFlash2-Drafter
vv_cli serve --model ./VibeVoice-ASR-AWQ-W4A16-ASYM --draft ./VibeVoice-ASR-DFlash2-Drafter --slots 4
flag
--draft <dir> this repository
--draft-block <n> rows of the drafted block the model checks per pass (the drafter drafts 8; without the flag: 4 or 8, whichever keeps more tokens per ms)
--draft-check exact|fast exact (default): the transcript without a drafter on the same --attn, bit for bit; fast: flashinfer attention and prefill projections for the check, the greedy transcript within rounding
--draft-quant int4|f16 how the runtime holds a BF16 drafter (INT4 by default)

It drafts for any VibeVoice-ASR (7B) checkpoint of the same model β€” microsoft/VibeVoice-ASR, scerz/VibeVoice-ASR-4bit β€” since it reads only the model's hidden states, embedding and LM head; it was trained on the AWQ checkpoint's transcripts, so that is where it drafts best. On NF4 (bitsandbytes) and INT8 weights only --draft-check fast pays: their decode step has no multi-row twin, so an exact check would cost a step per row, and vibevoice.c declines the drafter there with a pointer to fast.

Results

vibevoice.c b72be15 (branch dflash2), RTX 3090, VibeVoice-ASR-AWQ-W4A16-ASYM, greedy decoding, decode tokens per second:

plain drafted speedup tokens per block same transcript
20 held-out clips (AMI, VoxConverse, earnings calls, LibriSpeech, FLEURS) 148 tok/s 566 tok/s 3.84x 5.33 20/20
2-minute file, 2 speakers 151 tok/s 547 tok/s 3.63x 4.87 yes
32-minute file 123 tok/s 290 tok/s 2.36x 3.57 yes

Plain = the same binary without --draft, both with flashinfer attention, which --draft picks by default; --draft-check exact with the check width it measures. Against the default without a drafter (fa2): 3.82x on the 2-minute file, 2.73x on the 32-minute file.

The drafter

  • DFlash 2: 5 Qwen3-style layers (hidden 3584, 28/4 heads, intermediate 9472), 831M parameters, block 8.
  • KV injection: for every position the model has processed, the outputs of its layers 1, 7, 13, 19 and 25 are concatenated, projected (fc) and normalized; every drafter layer turns them into keys and values. The drafter reads the model's own view of the audio and of the transcript so far.
  • Two-tap dynamic convolution around attention and MLP (per-row kernels predicted from the row), and a candidate selector that re-ranks the model head's top 16 per row with a pairwise predecessor/successor term (rank 256).
  • The embedding and the LM head are the model's, frozen. The drafter scores a draft vocabulary of 32768 ids (draft_vocab): the ids its training transcripts use, plus every stop id.
  • Accepted drafts on 43 held-out clips' traces: 5.273 tokens per block of 8 (the selector's, with every earlier draft right).

Training

Self-distillation: the drafter learns what this model writes, timestamps and speaker ids included. tools/dflash in vibevoice.c holds the pipeline:

  1. ~390 hours of audio, 8228 training clips and 43 held out, from LibriSpeech, FLEURS (8 languages), SOVA, AMI, earnings calls and VoxConverse β€” audio only; the datasets' own text is never used.
  2. The model's own greedy transcripts of every clip (vv_dflash_data gen).
  3. Traces: every position's token, its role and the five tapped layers' outputs, replayed through the runtime (vv_dflash_data trace).
  4. train.py (PyTorch, flex attention): anchors at generated positions, rows 1..7 scored against the next 7 tokens with weights e^(-k/4); cross-entropy of the head plus the selector's; AdamW, cosine schedule, BF16.

Limits

  • vibevoice.c only: the checkpoint keeps the published DFlash 2 names, but the KV injection, the selector and the draft vocabulary are as vibevoice.c implements them (docs/DFLASH.md).
  • CUDA only (the CPU path and Metal decode without the drafter). Greedy only.
  • Audio the corpus covered little (Mandarin, Russian) keeps fewer drafts; the runtime then falls back to plain steps and stays near plain speed.

License

MIT, like VibeVoice.

Downloads last month
46
Safetensors
Model size
0.8B params
Tensor type
I32
Β·
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Ar4ikov/VibeVoice-ASR-DFlash2-Drafter

Finetuned
(1)
this model

Collection including Ar4ikov/VibeVoice-ASR-DFlash2-Drafter