<- Publications

A benchmark that checks the numbers audio language models state about a recording against the instruments that define them.

One two-talker mixture shown four times, for F0 mean, HNR, jitter and SNR: its spectrogram, the second talker's spans, the frames the instrument reads, and the reference decoder's decomposition of each value it states
One 9.44 s mixture in four columns. Rows are the mixture, the second talker's spans, the frames the instrument reads, and our reference decoder's 58-token decomposition. Each box is the shortest window holding half the reference's magnitude, and each peak a frame's height over an even split.

AcoustiClaim asks an audio language model to describe a recording, extracts every numeric claim from the text it writes, and scores each claim against the instrument that defines the quantity. The paper is under review at ICASSP 2027.

One recording decoded twice by the reference decoder. On a two-talker mixture it states SNR, SRMR, speaking rate and pauses but declines the pitch and voice-quality measures; on the clean twin of the same clip it states all ten quantities.

One recording decoded twice by our reference decoder, on a two-talker mixture and on its clean twin. On the mixture it declines the five voice quantities; on the twin it states all ten. Underlined: a value the parser recovered.

Ten quantities are scored: SRMR, SNR, speaking rate, pause count, pause rate, F0 mean, F0 SD, jitter, shimmer and HNR. They are measured on Libri2Mix two-talker mixtures and on AMI distant-microphone meetings. Each quantity is classed by where its reference can be read: in the signal the model hears, in a mixing parameter, on a clean stem the model never hears, or on the mixture outside the detected overlap. That way, every score says which instrument it was taken against.

What’s in the release

  • The parser and the scorer. They read quantities and units out of prose, and score each cell by Spearman rank correlation and by normalized absolute error.
  • The panel. Five audio language models (Qwen2-Audio-7B, Qwen2.5-Omni-7B, Granite-Speech-8B, Audio-Flamingo-3 and Gemini-3.8-Flash), each prompted five ways on 839 Libri2Mix mixtures and 859 AMI clips.
  • A reference decoder’s outputs. A frozen speech encoder feeds an instruction-tuned language model, with an auxiliary head that withholds a quantity when its own variance is largest.

Radar chart of Spearman rank correlation per quantity on Libri2Mix, comparing five audio language models and the reference decoder

Spearman rank correlation per quantity on the 839 Libri2Mix panel clips, best prompt per cell. The dashed ring is the reporting bar of 0.3.

Some figure elements are BioRender content. Created in BioRender. Lin, S. (2026).