EMNLP 2026 main conference

HEAR Who Said WhatUnlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding

Dongwook Lee1   Sangkwon Park1   Eunwoo Song2   Che Hyun Lee1
Youngho Cho1   Junho Kim1   June Young Yi1   Heeseung Kim3†   Sungroh Yoon1†
1Seoul National University 2Yonsei University 3University of Seoul
† denotes corresponding authors
📜 arXivTBA 🤗 Dataset 🧠 Model 💻 Code 🎧 Hear a hard negative
Not published yet, links go live with the release.
One clip · four speakers · each bar is one diarization box
The question

Do speech language models actually tell voices apart?

When several people are talking, answering well means knowing who said something, not just what was said. We test whether that ability is really there.

HEAR is a check on a foundational capability: hold the words fixed, change the voice, and see whether the answer follows.

Counterfactual voice grounding

Take one utterance out of a real conversation and put it back, same words, different voice. Nothing that could be read off the transcript has changed. Only the acoustics did, and the right answer moved with them.

The swap, drawn one slot, two voices

What a counterfactual pair guarantees

  1. 1The transcript and the temporal structure of the swapped clip are preserved relative to the original.
  2. 2Speaker similarity between the re-recorded utterance and its reference voice exceeds 0.7 (ECAPA-TDNN), so the new voice really is the target speaker's.
  3. 3Whisper-large-v3 transcribes the synthesised utterance at WER = 0 against the reference text, so the linguistic content is untouched.
  4. 4The answer is required to differ between the two versions. A model that ignores acoustic identity answers identically, and necessarily fails.
  5. 5Pairs are kept only if human annotators verify that the synthesised utterance matches the intended target speaker.

What a hard negative sounds like

A CASH-60K sample, from the Attribution category (Content → Voice): one utterance is re-recorded in another speaker's voice, so the correct answer moves with it.

Loading examples…
The benchmark

HEAR: three capabilities, in order

Following Erber's auditory hierarchy, HEAR breaks “who said what” into the foundations it actually rests on. You cannot bind an utterance to a voice you cannot tell apart, and you cannot reason over speakers you cannot bind.

Discrimination627 q
“Are these the same voice?”
VCVoice Cardinality. How many unique speakers are in this clip?
VLVoice Localization. Given a reference voice, where does that speaker appear, or not appear?
VCDVoice Change Detection. Where does the speaker turn over, or where do two voices overlap?
Attribution608 q
“Whose voice said this?”
CVAContent→Voice. Given a transcribed utterance, pick its speaker from five candidate voices.
VCAVoice→Content. Given a reference voice, pick which of five utterances it spoke.
Reasoning1,160 q
“What follows from who said what?”
IRIdentity. Which other utterance came from the same person?
TRTemporal. First or last utterance by a speaker; a turn located relative to someone else's.
QRQuantitative. How many speakers said X; who ranks n-th by total speaking time.

Every Reasoning item is a counterfactual pair and is counted twice, once as the original and once as the swapped variant, so that the pair-level metric has both halves. Overlap-annotated variants exist for every task except VC.

The public release, at a glance

887
Audio clips
2,395
Verified questions
3
Source corpora
22.8h
Of multi-party audio
5
Options per question
Clips by corpus
VoxMM · 360
ICSI · 286
AMI · 241
VoxMM: in-the-wild web video, 11 genres, 245 speakers, avg 87.8s ICSI: research meetings, 21 speakers, avg 94.9s AMI: scenario meetings, 24 speakers, avg 97.4s

Clips run 30–150 s and average 4.05 speakers (2–12). Questions are five-way multiple choice, generated by a rule-based pipeline and then filtered by eight expert annotators for solvability, answer uniqueness, voice-identity preservation and transcript–option consistency.

Listen to the benchmark

Three clips from the release, one per corpus, with eleven of their questions. Click the waveform, a speaker lane or a transcript line to play from that point. On a Reasoning question, open One slot, two voices and hear the same second in both voices.

This browser cannot decode either embedded audio format. Everything else on the page works; playback needs a current version of Chrome, Firefox, Edge or Safari.
Sample viewer · audio embedded, no server HEAR · 887 clips · 2,395 questions
Clip
click the waveform, a lane, or a transcript line to play from there
– –
Clip audio
0.0s / 0.0s
Speed 1.0x
Same clip, other voice
[ press play ]
Questions on this clip
Results

Twenty models, and where the floor is

Accuracy per item, with the paired score in parentheses on the three Reasoning columns, correct only when the model answers both the original clip and its voice-swapped counterfactual correctly. The gap between the two numbers is the size of the semantic shortcut.

Discrimination Attribution Reasoning  (paired)
ModelSize VCVCDVL CVAVCA IRQRTR Avg.

Table 1 of the paper, on the released benchmark (2,395 questions). Avg. in parentheses recomputes the overall mean with the paired Reasoning scores. “Random Choice” is the five-way floor.

How much is hallucinated

Reasoning accuracy scored per item, then per pair. The drop is the share of answers that were never grounded in the voice: the model was right about the words and guessed the speaker.

per item per pair

Zero-shot, on tasks it never trained on

Three OOD benchmarks where the query cannot be answered without knowing who spoke. A2R never saw any of them during training.

ModelWDYLGAOKAOFTS
Random Choice50.050.023.8
Qwen3-Omni-Instruct 30B61.365.629.2
Qwen3-Omni-Thinking 30B67.154.817.2
A2R, CASH-60K without hard negatives95.078.540.5
A2R (CASH-60K)97.788.248.2

WDYL = What Do You Like? (identity-critical questions); GAOKAO = listening-exam dialogue QA; FTS = Find the Spy, a 3–6 player social-deduction game. Removing the voice-swapped hard negatives from CASH-60K costs HEAR reasoning pair accuracy 60.0 → 43.0.

Release

Citation & availability

The benchmark, the CASH-60K corpus and the A2R checkpoint will be released together. HEAR and CASH-60K are released under CC BY-NC 4.0, and the source corpora keep their own terms; the underlying VoxMM videos remain the property of their uploaders. The base model, Qwen3-Omni-30B-A3B-Instruct, is Apache 2.0.

🤗
HEAR & CASH-60K
2,395 benchmark questions, plus the 59,762-query training corpus in CASH-60K/.
🧠
A2R
The merged 30B checkpoint. Load it straight into transformers or vLLM.
💻
Evaluation code
Inference and scoring for 25 speech language models, and for your own audio task.
 
@inproceedings{lee2026hear,
  title     = {HEAR Who Said What: Unlocking Speaker-Attributed Reasoning
               via Counterfactual Voice Grounding},
  author    = {Lee, Dongwook and Park, Sangkwon and Song, Eunwoo and
               Lee, Che Hyun and Cho, Youngho and Kim, Junho and Yi, June Young and
               Kim, Heeseung and Yoon, Sungroh},
  booktitle = {TO BE UPDATED ON PUBLICATION},
  year      = {2026}
}

Venue, pages and DOI are placeholders until the proceedings entry exists.