Note 2026-W42 · Places six systems at 427 pauses and 482 turn ends from 176 unscripted Seamless Interaction conversations. GPT Realtime, Gemini 3.1 Flash Live and Grok Voice respond at almost every pause, at about 1 to 4 s with silence endpointers and about 8 s with semantic turn detection. PersonaPlex and Moshi alone regularly stay silent. Text-history recall is 41 to 60% for hosted systems against 17 to 22% for full-duplex models (LLM judge). Cites TurnBench. Fullduplex is published by oto, which s

Note 2026-W42 · DyaFDB has two full-duplex models converse directly under assigned roles with cooperative or conflicting goals, and an external judge scores both sides offline. Four tasks, 140 scenarios and 7,560 conversations across six self- and cross-play pairings of PersonaPlex, MiniCPM-o and Raon-SpeechChat. Each model's behaviour reshapes its partner's; in the contested-channel task Raon holds 0.75 of exclusive speaking time against MiniCPM-o and 0.62 against PersonaPlex. Scenarios and protocol

Note 2026-W42 · Two PersonaPlex-7B instances converse on a shared clock, scored with the same floor-transfer rule as Switchboard. Timing is coupled (re-pairing speakers destroys it), but the floor changes hands at a median of 400-560 ms against 137 ms for humans, and the last 120 ms of the partner's turn holds 1% of model transfers against a tenth for humans. Delaying one channel shifts the response one-for-one, consistent with a reactive wait rather than turn-end projection.

Note 2026-W42 · RL framework that factorizes a full-duplex text policy into a control policy choosing 'pad', 'epad' or 'con' and a content policy that picks a token only on 'con'. Timing advantages go to the control factor through event-causal masks and an LLM-judge semantic advantage goes to the content factor. Across three Moshi seeds on Full-Duplex-Bench v1 it lowers takeover rates at user pauses and backchannel opportunities and shortens post-interruption latency relative to GRPO.

mistralai/Voxtral-Mini-4B-Realtime-Arabic

Automatic Speech Recognition • 4B • Updated 13 days ago • 1.73k • 47

Note 2026-W42 · Apache-2.0 streaming speech recognizer fine-tuned from Voxtral-Mini-4B-Realtime-2602 for Modern Standard Arabic and Maghreb, Gulf, Nile and Levantine dialects, with code-switching. Self-reported average CER of 8.82% across seven benchmarks at a 480 ms delay, within 0.91 points of offline Voxtral Transcribe Arabic (7.91%). Two benchmarks, ISMA and Darija in the Wild, are new and co-built with MTNRA; normalization code is published. Created 2026-10-08 with no launch post.

Note 2026-W42 · 2,100 hours of Mandarin multi-party conversation in Tutoring and Social Chat, with 2 to 4 synchronized participant-level tracks per session. SmoothConv is 100 human-verified hours with 3.33% overlap; DuplexConv is 2,000 automatically annotated hours. The SmoothConv Benchmark covers speech separation, multi-speaker ASR and a four-state turn-detection set (complete, incomplete, backchannel, wait). Stated as public on GitHub; no data licence stated.

Note 2026-W41 · Benchmark of 120 interactive professional workflows across ten archetypes, each run in a stateful Voice Workbench with typed tools, authorization constraints, a gold final artifact and a simulated user with pre-compiled speech. Success requires the correct terminal state, a valid process, completed actions and a valid artifact. Across GPT-Live-1, Gemini-3.8-Live, Grok-Voice-Think-2.0, Step-Audio3 and GPT-realtime-2.1, none exceeds 25% Pass@1 and the best Reliable@3 is 10.8%. Stateful

Note 2026-W41 · VAmoS Energy: 100 utility-billing calls where each caller makes two to four requests and the agent uses sixteen tools backed by a stateful Stripe billing twin and the Apache Fineract loan engine, with access blocked until verification. An LLM verifier agrees with a code verifier on 99.1% of checks. Across fourteen voice stacks and three repeats, completion ranges from 17.3% to 44.7%; Grok Voice leads. Background television cuts pooled completion from 38.7% to 8.6%. The simulated calle

Note 2026-W41 · Extends the Moshi paradigm to long-horizon, multi-party English-Chinese full-duplex dialogue. Releases 57.6k hours of synthetic training data (MultiTalkPT, MultiTalkFT) with controllable participant count, overlap, backchannels, interruptions and addressee shifts, plus MultiTalkBench, built from real recordings averaging 32.6 minutes. The authors' bilingual model substantially outperforms Moshi, MiniCPM-o-4.5 and Qwen3-Omni-30B-A3B-Instruct on MultiTalkBench; the abstract gives no sin

Note 2026-W41 · Bilingual benchmark of six full-duplex behaviours (including interruption, yielding, proactive initiation, active silence and backchanneling) under Pre-session, In-session and No-explicit conditions. 1,290 English and Chinese streaming trials score 12 systems on Timing and Content, finding large variation and frequent mismatch between semantic quality and timing. Per-author affiliation mapping is not stated. Fullduplex is published by oto; the No-explicit backchannel subset uses a sub

IndexTeam/Index-Echo-S2ST-9B

Updated 7 days ago • 501 • 20

Note 2026-W41 · Speech-to-speech translation that keeps the source speaker's voice: Chinese or English in, English, Spanish, Japanese or Chinese out. A frozen Index-Echo-S2TT backbone feeds a learned Hidden2CV mapper and CosyVoice3, trained with differentiable reward optimization. Apache-2.0, 9B and 2B, plus S2TT siblings; technical report arXiv:2609.40181. Vendor-reported: against a matched S2TT plus CosyVoice3 pipeline on in-house dubbing data, the 9B lowers mean content error in four of six direct

Note 2026-W40 · Argues that whether a delayed reply or an anticipatory overlap is appropriate depends on the speaker's latent intent, which binary fixed-window metrics ignore. TACT has 9,728 episodes and 73.2 hours from five dyadic corpora, each with dialogue history, a per-speaker memory profile and an annotator-derived posterior over six intent classes. Scoring uses intent-conditioned timing kernels fitted to human floor-transfer offsets. Across eleven systems the best scores 0.47 against a human t

Note 2026-W40 · 1,152 short multi-party, multi-turn audio scenes split evenly between Mandarin and English, each with participant context, expected tool calls and atomic rubrics for multi-speaker memory, instruction following and reasoning. The strongest configuration passes every rubric on 66.8% of English and 54.5% of Mandarin cases; the strongest open-weight one on 34.0% and 19.3%. Open-weight models are bottlenecked by the multi-speaker audio front end; frontier systems still fail speaker-scoped

Note 2026-W40 · Extends tau-Voice to Spanish, Brazilian Portuguese, Hindi, Korean and Mandarin with native-speaker review of generated language and speech. Across 4,500 full-duplex calls and five voice configurations, Spanish, Portuguese and Hindi stay within 3.2 task-completion points of English, while Korean and Mandarin fall 14.7 and 8.4 points. Korean systems miss more responses, Mandarin systems interrupt more, and both struggle with tools and entities. Grok leads task completion but scores lowe

Note 2026-W40 · Pairs full-duplex voice with asynchronous task execution. A Frontend Agent runs the dialogue and chooses between direct tool use and delegation, a Backend Agent does delegated work in a separate context, and an Orchestration Runtime tracks task state, requests input or authorisation, and schedules results back into the conversation. Interruption is separated from cancellation, and completion from delivery. On an in-house cockpit benchmark of 134 cases, mixed execution reaches 91.04% t

Edge0/Audio8-ASR-Infinite

Automatic Speech Recognition • 4B • Updated about 1 month ago • 39.5k • 873

Note 2026-W40 · A preview created 21 September that reached 1,896 likes and about 26,700 downloads by 1 October. A Voxtral Realtime 4B causal audio tower feeds a Qwen2.5-3B decoder in a delayed-streams design, with a selectable 80, 120 or 160 ms clock, a configurable 240 to 560 ms transcription delay, and a rolling KV cache that keeps memory and latency constant for audio of any length. Semantic VAD heads aim to separate thinking pauses, stuttering and real end of turn. Chinese and English only. No w

Note 2026-W40 · A weakly labelled corpus crawled from YouTube videos uploaded under a Creative Commons licence, with no overlap with YODAS v1 or v2: about 1.1 million hours of 48 kHz multi-channel audio in 147 languages, released under CC BY 3.0. About 75% has transcripts with sentence-level timestamps, 95% of those also word-level; around 60% of non-English audio has English translations. 22 languages exceed 10,000 hours and 73 exceed 5,000. The authors call it the largest open speech dataset to dat

nvidia/Nemotron-3-Diarization

Voice Activity Detection • 99.2M • Updated about 1 month ago • 118k • 805

Note 2026-W40 · The August preview shipped under an evaluation-only licence that barred production use and publishing results without NVIDIA's consent. NVIDIA announced the full release on 23 September under OpenMDW 1.1, which grants use without restriction provided the licence and notices are kept, with a litigation termination clause. The 100M-parameter model handles up to eight speakers and streams with input buffers down to 80 ms (0.32 s recommended). NVIDIA reports first place on VoiceArena's di

Note 2026-W39 · A duplex speech-to-text frontend learns to emit a delegation token and forward streaming transcripts to a text LLM backend for tool calls; results come back through a lightweight prefill-and-repeat step and out via streaming TTS. Turn-taking, interruption handling and latency are largely preserved because the frontend barely changes. Single-turn evaluation: 92 to 97% tool-call recall and 81.2% accuracy rejecting irrelevant calls. With a Qwen3-235B-A22B backend it is competitive on Ful

Note 2026-W39 · The system report for the 11B model NVIDIA released on Hugging Face on 29 July. A streaming speech encoder and decoder-only LLM with parallel output streams for agent text and structured function calls, an RNN-T branch for user transcription, and a streaming TTS decoder. On Full-Duplex-Bench 1.0 it has the lowest pause-handling takeover rate among open-weight systems, 100% takeover after interruptions and 4.33/5 post-interruption quality; on FDB 1.5 it resumes after backchannels 93% o

Note 2026-W39 · Separates the reason to speak from the opportunity: context-matched English monologues where only the trigger sentence varies, ten conditions from turn-allocation rules, and compressed pauses so silence creates fewer openings. Across five model families, being addressed and silence are far more reliable triggers than false facts or hazards. Given the floor, only 0.14 to 0.15 of non-empty false-fact replies challenge the claim and 0.04 to 0.07 of hazard replies warn. Two authors overla

Note 2026-W39 · A paired diagnostic benchmark for Chinese full-duplex turn-taking. Each pair shares the same overlap transcript but has contrasting preceding context, one requiring Yield and the other Keep, plus off-talk examples for unnecessary yielding. Pair accuracy requires both members correct and gives no credit to a constant policy. Most full-duplex systems tested show a pronounced bias toward Yield, doing much better on interruptions than on backchannels, while one stays balanced. Interruptio

Note 2026-W39 · An audio-language foundation model built around a listen-converse-think-act loop: Deep Perception for acoustic intent, Seamless Duplex for synchronised streams handling pauses, backchannels and interruptions, and Think-While-Speaking, which runs private reasoning in parallel with spoken delivery. An integrated agent executes tools asynchronously. Reported: 73.0 macro on StepAudioChat in reasoning mode, 90.6 on MMSU, 98.9 on the Artificial Analysis Full-Duplex Bench, 56.0% task success

inclusionAI/Realtime-Venus

Any-to-Any • Updated about 1 hour ago • 547 • 121

Note 2026-W39 · Weights and custom Transformers code released 16 September under Apache-2.0: an audio-visual model and an audio-only model, both 9B and adapted from MiniCPM-o 4.5. A shared causal timeline carries user input, model output and delegation events; a dual-loop runtime lets the Harness run reasoning and tools asynchronously while the foreground keeps talking. Paper (arXiv 2609.13814): on Full-Duplex-Bench v1.5 the audio model answers 75% of interruptions and continues through 97%, 88% and

moondream/parakeet-redux

Automatic Speech Recognition • 0.1B • Updated Sep 23 • 20.2k • 302

Note 2026-W39 · NVIDIA's parakeet-tdt-0.6b-v3 with every encoder weight quantised to -1, 0 or +1: same architecture and tokenizer, 178 MB instead of 1.2 GB, 113 times real time on eight x86 CPU cores. Open ASR Leaderboard 6.55 WER against the original's 6.26; FLEURS 25 languages 10.56 against 11.62; TED-LIUM long-form 2.51 against 2.71; background noise across nine MUSAN conditions 9.04 against 6.72, the one clear regression. Runs through Moondream's Photon engine on AVX-512, NEON and Metal. Repo cre

Note 2026-W38 · Delivers fixed, request-independent spoken prompts through the user audio channel while a full-duplex model is answering, comparing fixed-delay interruption with refusal-triggered interruption keyed to the model's streaming text. Across four open models and 720 requests from AdvBench and HarmBench, fixed-delay interruption raises attack success on AdvBench to 40.3% for PersonaPlex and 48.7% for PersonaPlex-RL, up 33.8 and 39.3 points; the refusal-triggered policy reaches 35.6% and 48.

Note 2026-W38 · A controlled ablation of acoustic, prosodic and semantic signals for streaming end-of-turn detection with a lightweight trimodal classifier under identical training. Acoustic plus prosodic features give the best accuracy-latency balance: utterance F1 0.93 with 7.8% false alarms at 400 ms median latency. Adding text increases premature detections without improving performance; prosodic features separate the classes best and text representations overlap. Turn-taking is carried mainly by

Note 2026-W38 · Introduces Duplex Cue, an evaluation of the response humans use routinely and full-duplex benchmarks cannot express: continuing to speak while folding in what the listener just contributed, a missing word, a correction or a clarification. It separates listener intent (backchannel, collaboration, interruption) from speaker behaviour (continue, adapt within the turn, yield). A single-model case study uses 300 human-confirmed cues from unscripted English conversation, comparing recorded

Note 2026-W38 · A mixture-of-experts, LLM-based ASR system on the Qwen backbone trained on tens of millions of hours, aimed at production gaps rather than benchmarks: regional dialects, dynamic entities and hotwords, long-range context and disfluent spontaneous speech, through a single instruction-following interface. Supports 30 languages and 16 Chinese dialectal varieties across eight dialect regions. Submitted 7 September and revised 9 September. No public weights or repository for this version we

Note 2026-W38 · A general-purpose audio generator covering zero-shot TTS, voice design, vocals, sound effects, music, vibe speech and mixtures, built as a discrete autoregressive model over residual vector quantisation tokens rather than the diffusion-transformer approach of most recent general audio models. Its tokenizer represents audio at 12.5 Hz in a shared 16 x 2048 residual code space that jointly quantises semantic and waveform-level features. The backbone predicts the first codebook along tim

tencent/AuK

Text-to-Speech • Updated Sep 10 • 1.94k • 391

Note 2026-W38 · Open-sourced 9 September with a technical report (arXiv 2609.08936) the day before. One instruction-driven model covers five task families: speech generation, content editing, enhancement and separation, paralinguistic editing and acoustic editing, trained on about 3.03 billion instruction-audio instances and 1.95 million hours of supervision. A multimodal LLM conditions semantics, a VAE trained on speech, audio and music conditions acoustics, and a hybrid rectified-flow transformer g

Note 2026-W37 · A 3B diffusion transformer trained with flow matching on 480k hours of speech, then fine-tuned for cross-lingual dubbing, full-duplex dialogue synthesis and emotional dialogue synthesis. It runs on DAC-VAE latents mapping 48 kHz audio to 25 Hz, over 10x EnCodec's compression, and is alignment-free: alignment learned by cross-attention, no duration predictor. One-shot generation to about a minute, long-form via multi-diffusion. Reported to approach human recordings on short conversatio

Note 2026-W37 · Neural finite-state-machine dialogue puts turn-taking control tokens and response text on one causal tape under ordinary next-token prediction, keeping the base LLM's semantics at low fine-tuning cost. Its weakness has been synthetic text training data, since LLMs cannot simulate real acoustic timing. This work learns turn-taking from real human-human spoken dialogue and semantics from human-agent text, with a rule-based transformation that serialises recordings into FSM tapes without

Note 2026-W37 · A mechanistic trace of speaking-style information through Whisper-large-v2, Qwen2-Audio-7B-Instruct, Qwen2.5-Omni-7B and Chroma-4B on Expresso, using centered kernel alignment, leave-one-speaker-out probes, open-ended tone prediction and a content-prosody leakage metric. All four strongly encode style in the top third of the audio encoder, and all degrade it before the output. The projector changes geometry without removing information; decoders differ in how much style survives. Mode

Note 2026-W37 · Introduces Output Divergence Rate, the share of utterances where speech enhancement changes an LLM's intent classification relative to clean speech, benchmarked over five conditions on 2,974 SLURP clips through Whisper large-v3 and wav2vec2-large cascades. Every condition diverges significantly from zero. MetricGAN+ more than doubles ODR versus unenhanced noisy speech, 0.318 against 0.135, while improving PESQ; unmitigated echo reaches 0.836 through speaker substitution, a failure WER

microsoft/VibeVoice-ASR-Streaming-7B

Automatic Speech Recognition • 9B • Updated Sep 3 • 1.66k • 264

Note 2026-W37 · An LLM-based end-to-end model that produces who-said-what as speech arrives, interleaving fixed-size audio chunks, a small lookahead and previous text so no separate diarization stage is needed. The technical report (arXiv 2609.02812) claims the 7B has the lowest average WER/CER across five evaluation sets and the best or tied-best speaker attribution on 12 of 13 settings. Ten languages and user-supplied hotwords. Weights for both sizes and inference code are MIT. Topped the Hugging F

Note 2026-W37 · Six months of real-world usage from more than 500 users of a smartwatch health assistant, yielding 3,030 anonymised utterances that triggered a fallback: noisy audio, transcription errors, ambiguous requests, incomplete utterances and unintended activations. The paper contributes an operational taxonomy, the annotated dataset, and a comparison of classifiers under deployment constraints, finding that lightweight embedding-based classifiers beat larger generative models on most tasks a

Note 2026-W37 · 20.0 hours across 58 sessions and seven collaborative tasks (Spot the Difference, Photo Talk, Describe-and-Draw Portrait, Tangram Direction, Consensus Ranking, Hiring Decision, Voice-to-Form), 48 kHz channel-separated FLAC as WebDataset shards, with 11,025 timestamped interface events, per-event visibility, task stimuli and reference answers where the task has one. No transcripts. CC BY 4.0, gated with manual approval. Published 28 August, inside last week's window, and missed there.

Note 2026-W36 · A 30-hour hand-labelled corpus of dyadic conversation plus a fixed protocol for scoring end-of-turn and interruption detection, conversation type controlled across six styles, every dialogue triple-annotated at Fleiss's kappa 0.78. Fourteen systems were scored. End-of-turn recall is stable across types; interruption false positives concentrate in backchannel-dense talk. No system is simultaneously fast, high-recall and low on false positives. Disclosure: the training split is oto data

Note 2026-W36 · Two omni-modal models converse in native audio with no external speech recognition or synthesis and no API boundary, over the unmodified tasks, tools and success checks of an established text agentic benchmark, so interaction modality is the only variable and the training loop stays local and differentiable. Existing setups either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow, or stay in text and can measure voice agents without improving them. The re

Note 2026-W36 · Models how gaze, speech and perceived interpersonal closeness signal floor changes in free four-person dialogue, using the GaMMA corpus and interpretable logistic regression over behaviourally motivated features extracted before each turn-taking event, classifying outcomes as gaps or overlaps. Most turn-taking work this window is dyadic and audio-only; this is the multi-party, multimodal case, and it deliberately trades detector accuracy for features a designer can reason about.

BreezeBlue/Breeze-TTS-2

Text-to-Speech • 3B • Updated Sep 9 • 14.1k • 767

Note 2026-W36 · A text-to-speech release covering voice cloning, voice design and voice direction, which reached 215 likes and about 1,800 downloads in its first week and now leads the Hugging Face text-to-speech trending list. The licence is split and worth reading before integration: source code is Apache-2.0, but the model weights, any derivative models, and self-hosted outputs are restricted to research and non-commercial use. BreezeBlue is a new name in this digest and has been added to the org

Note 2026-W35 · 480 persona-grounded scenarios hold the task fixed and vary whether the user's concern is stated in words or carried only in prosody, with objectively checkable outcomes. Giving the model the audio on top of the transcript moves the optimal-solution rate from 14.6% to 15.3%. Forcing it to first write the inferred concern into text takes the same models to 39.6%, against 40.7% for ground-truth state. The prosody is recoverable, and it still does not reach the action unless something ma

Note 2026-W35 · Synthesises from uncertain token prefixes instead of waiting for a sentence, using uncertainty-aware buffering and carrying decoder state across segment boundaries. Reported at 15.8ms median time to first token for a single request and 260.8ms at 128 concurrent. Five of its seven authors also wrote X2-Turn, the streaming turn-state model from last week, so one company is now assembling a real-time voice stack part by part without training an end-to-end duplex model at any point.

Note 2026-W35 · Instead of sweeping blindly through distortions, this uses cheap structural probes to locate the domain a watermark is embedded in, then applies a single attack matched to that domain, and reports a threshold-free fragility score per scheme. It needs no training and no access to the watermarking model. Anyone planning to satisfy a marking obligation with a neural audio watermark should read it before treating that watermark as the compliance artefact.

Note 2026-W35 · Adds real-time predicted backchannels and head nodding to a voice-cloned avatar and measures the effect in a within-subjects study of 35 people. Perceived attentiveness, the sense of talking with the real person, and co-presence all improve significantly. The argument is that a duplex agent feels present because of how it listens, not how well it speaks, which is a case for spending latency budget on the listening side.

Note 2026-W35 · Three behavioural probes, covering reference disagreement, masked-number recovery, and orthographic switching, show leading open ASR models reproducing verbatim benchmark reference spans even when the audio contradicts them, masks them, or leaves them ambiguous. The behaviour can be steered with a low-rank direction, which makes it a learned policy rather than an artefact. Anyone choosing a backbone off a WER leaderboard is reading a number that partly measures memorisation.

FireRedTeam/FireRedAudio

Updated Aug 26 • 51

Note 2026-W35 · One backbone with two decoupled continuous paths, an audio encoder for understanding and a RedAE path for generation, covering ASR, audio understanding, zero-shot and instruct TTS, semantic and acoustic speech editing, and temporal grounding over recordings up to an hour. Weights are up under Apache-2.0. The benchmark claims on MMAU, MMSU, Seed-TTS-Eval and InstructTTSEval are self-reported and the linked paper is still a placeholder, so treat the scores as unverified.