Transcribing lyrics from a song, and why it goes wrong

Why singing is a different problem from speech

Whisper was trained on hundreds of thousands of hours of audio, nearly all of it people talking. Singing changes most of what the model learned to rely on. Vowels are stretched across beats, the pitch contour carries melody instead of meaning, consonants are softened to hold a note, and the instruments sit in the same frequency band as the voice and mask it.

The model also hears everything through a 16kHz mono spectrogram, which is fine for speech and throws away some of what separates a voice from a guitar. Reverb, which most mixes have plenty of, smears the word boundaries the model needs in order to tell where one word ends.

So the ranking is predictable. Rap and spoken-word come out best, because they are closest to speech. A singer-songwriter over a single acoustic instrument does reasonably. A dense mix, a choir, or anything operatic comes out worst, and the model's failures there are not random.

The three ways it fails

Each of these looks different in the transcript, and knowing which one you are looking at saves time.

• Nothing. The model decides a stretch is music or silence and emits no text for it, so whole verses are missing and the timestamps jump. This is the honest failure and the easiest to work with
• Invented lines. The model produces fluent, lyric-shaped text that is not what was sung. Over an instrumental break it can produce a sentence with no vocal under it at all. These read plausibly, which is what makes them dangerous, and the only way to catch them is to listen
• Loops. The decoder gets stuck and repeats a phrase, sometimes for the rest of the song. It is Whisper's characteristic failure over music, and it is mechanical enough to detect, so this editor flags any cue that repeats itself and you can delete the run in one pass
• A fourth case that is not a failure but is worth knowing: a well-known song sometimes comes out right for the wrong reason. The lyrics were in the model's training data and it completes them from memory rather than from the audio, which means a cover with changed words can come out as the original

What helps

One change dominates everything else, and the rest are small.

• Separate the vocal first. A stem separator such as Demucs, which is open source and runs on your own machine, pulls the voice out of the mix, and a transcript of the isolated vocal is a different quality of result from a transcript of the full song. Do this before anything else
• Name the language instead of leaving it on automatic. Detection reads only the first 30 seconds, and most songs open with an instrumental intro that tells it nothing
• Use the larger of the two models. It is the bigger download and the slower run, and on music the difference is worth it
• Trim to the sung section if the recording has long instrumental passages, which removes the stretches where the model is most likely to invent or loop
• If you have a demo, a rehearsal take or an a cappella version, transcribe that rather than the master. The words are the same and the mix is not in the way

What a lyric transcript is for

Nobody needs a speech model to tell them the words to a released song. The lyrics are on the sleeve, and a search finds them. What the transcript provides is timing: each phrase with the second it starts, which is the tedious part of making a lyric video and the part a listener notices when it is wrong.

So the useful workflow is to take the timed draft, correct the words against the real lyrics while keeping the times, and burn the result into the video. The same renderer draws the preview and the output, so what you see over the clip is what the file will contain. The reading-speed flags will fire on a fast chorus, and for lyrics that is information rather than an instruction. People read a lyric they already know faster than they read a stranger's sentence.

Where not uploading matters here is the unreleased song. A demo of something that is not out yet is exactly the file its writer does not want on a transcription service's disk, and here it never leaves the machine it was recorded on.