Diarizing Speech with WhisperX and NVIDIA’s NeMo

Welcome to Chapter 8, where we will explore the world of speech diarization. While Whisper has proven to be a powerful tool for transcribing speech, there’s another crucial aspect of speech analysis that can significantly enhance its utility – speaker diarization. By augmenting Whisper with the ability to identify and attribute speech segments to different speakers, we open a new realm of possibilities for analyzing multispeaker conversations. This chapter will explore how Whisper can be integrated with cutting-edge diarization techniques to unlock these capabilities.

We will start by exploring the evolution of speaker diarization systems, from the limitations of early approaches to the transformative impact of transformer models. Through practical, hands-on examples, we’ll preprocess audio data, transcribe speech with Whisper, and fine-tune the alignment between transcriptions and the original audio.

...