DiCoP v2 β€” Diarization-Conditioned Parakeet

Long-form target-speaker ASR built on Parakeet-TDT-0.6b-v3. Given a recording and its diarization, it transcribes one speaker at a time, and decodes a whole meeting per speaker in one pass. It is the backbone of Pandemonium, described in section 2.2 of the paper, and comes in two variants:

variant folder head RTFx use it for
TDT tdt/ Parakeet's TDT transducer, fine-tuned 269 accuracy (the "DiCoP" row of the paper's Table 1)
CTC ctc/ Parakeet's joint, frozen, used as a frame-synchronous CTC head 405 speed: 1.5Γ— faster, 0.6–1.5 cpWER worse

RTFx: oracle diarization, greedy decoding, whole recordings (Fig. 2 of the paper). For better accuracy use Pandemonium, which adds cross-speaker attention and diarization-mask refinement on top of the CTC variant and more training data.

How it works

The diarization decides, for the target speaker, whether each frame is silence, target speech, other speakers or overlap (an STNO mask). Every Conformer layer applies a small learned transform per class to its input, a frame-level diarization-dependent transformation (FDDT), as in DiCoW. There is no segmentation, no speaker embedding and no separation front end. Local attention lets the encoder take a whole meeting at once.

Backbone Parakeet-TDT-0.6b-v3 FastConformer encoder, 24 layers, d = 1024, 8Γ— subsampling (80 ms frames)
Attention local [256, 256] frames + 4 global tokens
Parameters β‰ˆ0.7 B
Input 16 kHz mono audio + a diarization (RTTM, or Lhotse cuts with supervisions)
Output one time-stamped word sequence per speaker

Training

Both variants are trained on the original SE-DiCoW recipe data (1.2k h, 30 s windows), starting from Parakeet-TDT-0.6b-v3: the single-distant-microphone training sets of NOTSOFAR-1 and AMI, noisy Libri2Mix and Libri3Mix (train-clean-100/360), and LibriSpeechMix-like 3-speaker mixtures. Best epoch by validation cpWER: TDT epoch 4 (0.1939), CTC epoch 6 (0.2019). The configuration stored in each checkpoint is included as model_config.yaml.

Results

cpWER [%], whole recordings, greedy decoding (TDT variant, Table 1 of the paper):

diarization NSF AMI ICSI Mx6 SBC DiPCo Avg
oracle 17.3 14.5 14.4 11.9 16.0 26.1 16.7
pyannoteAI Precision-3 23.2 16.9 18.0 12.1 19.4 28.9 19.8

NSF: NOTSOFAR-1 eval (single device); Mx6: Mixer 6; SBC: Santa Barbara Corpus. Pandemonium reaches 14.7 / 19.2 on the same sets.

Usage

Inference runs from the code repository (infer.py), with the config of the variant's head and Parakeet-v3 supplying the tokenizer:

hf download BUT-FIT/DiCoP-v2 --local-dir DiCoP-v2

# TDT
python infer.py --rttm rttms/ --audio-dir audio/ --output hyp.stm \
    --checkpoint DiCoP-v2/tdt/cp_wer=0.1939-epoch=4.ckpt \
    --config conf/dicop.yaml -O init_from_pretrained=nvidia/parakeet-tdt-0.6b-v3

# CTC
python infer.py --rttm rttms/ --audio-dir audio/ --output hyp.stm \
    --checkpoint DiCoP-v2/ctc/cp_wer=0.2019-epoch=6.ckpt \
    --config conf/dicop_ctc.yaml -O init_from_pretrained=nvidia/parakeet-tdt-0.6b-v3

--cuts cuts.jsonl.gz takes a Lhotse cutset instead of RTTMs and audio. The output is an STM with one line per word and speaker.

Files

file contents
tdt/cp_wer=0.1939-epoch=4.ckpt TDT weights (PyTorch Lightning checkpoint: state_dict + hyper_parameters, fp32, 2.8 GB)
ctc/cp_wer=0.2019-epoch=6.ckpt CTC weights (same format)
*/model_config.yaml the configuration stored in the checkpoint
*/tokenizer.model SentencePiece tokenizer, shared with Parakeet-TDT-0.6b-v3

Training data and licences

source licence
Parakeet-TDT-0.6b-v3 (initialization) CC-BY-4.0
NOTSOFAR-1 CC-BY-4.0
AMI CC-BY-4.0
LibriSpeech (Libri2Mix, Libri3Mix, LibriSpeechMix-like mixtures) CC-BY-4.0

No training audio or transcripts are redistributed with the model. The weights are released under CC-BY-4.0.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for BUT-FIT/DiCoP-v2

Finetuned
(143)
this model

Datasets used to train BUT-FIT/DiCoP-v2