Instructions to use BUT-FIT/DiCoP-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NeMo
How to use BUT-FIT/DiCoP-v2 with NeMo:
import nemo.collections.asr as nemo_asr asr_model = nemo_asr.models.ASRModel.from_pretrained("BUT-FIT/DiCoP-v2") transcriptions = asr_model.transcribe(["file.wav"]) - Notebooks
- Google Colab
- Kaggle
DiCoP v2 β Diarization-Conditioned Parakeet
Long-form target-speaker ASR built on Parakeet-TDT-0.6b-v3. Given a recording and its diarization, it transcribes one speaker at a time, and decodes a whole meeting per speaker in one pass. It is the backbone of Pandemonium, described in section 2.2 of the paper, and comes in two variants:
| variant | folder | head | RTFx | use it for |
|---|---|---|---|---|
| TDT | tdt/ |
Parakeet's TDT transducer, fine-tuned | 269 | accuracy (the "DiCoP" row of the paper's Table 1) |
| CTC | ctc/ |
Parakeet's joint, frozen, used as a frame-synchronous CTC head | 405 | speed: 1.5Γ faster, 0.6β1.5 cpWER worse |
RTFx: oracle diarization, greedy decoding, whole recordings (Fig. 2 of the paper). For better accuracy use Pandemonium, which adds cross-speaker attention and diarization-mask refinement on top of the CTC variant and more training data.
How it works
The diarization decides, for the target speaker, whether each frame is silence, target speech, other speakers or overlap (an STNO mask). Every Conformer layer applies a small learned transform per class to its input, a frame-level diarization-dependent transformation (FDDT), as in DiCoW. There is no segmentation, no speaker embedding and no separation front end. Local attention lets the encoder take a whole meeting at once.
| Backbone | Parakeet-TDT-0.6b-v3 FastConformer encoder, 24 layers, d = 1024, 8Γ subsampling (80 ms frames) |
| Attention | local [256, 256] frames + 4 global tokens |
| Parameters | β0.7 B |
| Input | 16 kHz mono audio + a diarization (RTTM, or Lhotse cuts with supervisions) |
| Output | one time-stamped word sequence per speaker |
Training
Both variants are trained on the original SE-DiCoW recipe data (1.2k h, 30 s windows), starting from
Parakeet-TDT-0.6b-v3: the single-distant-microphone training sets of NOTSOFAR-1 and AMI, noisy Libri2Mix and
Libri3Mix (train-clean-100/360), and LibriSpeechMix-like 3-speaker mixtures. Best epoch by validation cpWER:
TDT epoch 4 (0.1939), CTC epoch 6 (0.2019). The configuration stored in each checkpoint is included as
model_config.yaml.
Results
cpWER [%], whole recordings, greedy decoding (TDT variant, Table 1 of the paper):
| diarization | NSF | AMI | ICSI | Mx6 | SBC | DiPCo | Avg |
|---|---|---|---|---|---|---|---|
| oracle | 17.3 | 14.5 | 14.4 | 11.9 | 16.0 | 26.1 | 16.7 |
| pyannoteAI Precision-3 | 23.2 | 16.9 | 18.0 | 12.1 | 19.4 | 28.9 | 19.8 |
NSF: NOTSOFAR-1 eval (single device); Mx6: Mixer 6; SBC: Santa Barbara Corpus. Pandemonium reaches 14.7 / 19.2 on the same sets.
Usage
Inference runs from the code repository (infer.py), with the
config of the variant's head and Parakeet-v3 supplying the tokenizer:
hf download BUT-FIT/DiCoP-v2 --local-dir DiCoP-v2
# TDT
python infer.py --rttm rttms/ --audio-dir audio/ --output hyp.stm \
--checkpoint DiCoP-v2/tdt/cp_wer=0.1939-epoch=4.ckpt \
--config conf/dicop.yaml -O init_from_pretrained=nvidia/parakeet-tdt-0.6b-v3
# CTC
python infer.py --rttm rttms/ --audio-dir audio/ --output hyp.stm \
--checkpoint DiCoP-v2/ctc/cp_wer=0.2019-epoch=6.ckpt \
--config conf/dicop_ctc.yaml -O init_from_pretrained=nvidia/parakeet-tdt-0.6b-v3
--cuts cuts.jsonl.gz takes a Lhotse cutset instead of RTTMs and audio. The output is an STM with one line per
word and speaker.
Files
| file | contents |
|---|---|
tdt/cp_wer=0.1939-epoch=4.ckpt |
TDT weights (PyTorch Lightning checkpoint: state_dict + hyper_parameters, fp32, 2.8 GB) |
ctc/cp_wer=0.2019-epoch=6.ckpt |
CTC weights (same format) |
*/model_config.yaml |
the configuration stored in the checkpoint |
*/tokenizer.model |
SentencePiece tokenizer, shared with Parakeet-TDT-0.6b-v3 |
Training data and licences
| source | licence |
|---|---|
| Parakeet-TDT-0.6b-v3 (initialization) | CC-BY-4.0 |
| NOTSOFAR-1 | CC-BY-4.0 |
| AMI | CC-BY-4.0 |
| LibriSpeech (Libri2Mix, Libri3Mix, LibriSpeechMix-like mixtures) | CC-BY-4.0 |
No training audio or transcripts are redistributed with the model. The weights are released under CC-BY-4.0.
- Downloads last month
- -
Model tree for BUT-FIT/DiCoP-v2
Base model
nvidia/parakeet-tdt-0.6b-v3