Project 23: Speech-to-Text with Whisper

Introduction

OpenAI's Whisper changed speech-to-text from a paid-API affair into a pip install. It is open-source, runs locally, handles 99 languages, and punches through accents and background noise that older engines choke on. This app wraps it in a friendly UI: upload an audio file or record one in the browser, get a transcript — with optional timestamps — in seconds.

It is the natural output side of the audio recorder, and the reverse of the text-to-speech app: together they form a full audio round-trip.

Features

Two inputs — upload WAV/MP3/M4A or record via microphone component.

Model size selector — tiny (fast) to medium (accurate), with honest speed tradeoffs.

Plain and timestamped modes — paragraphs or [00:12] cues for navigation.

Language handling — auto-detect or force a language.

TXT/SRT-style download — take transcripts anywhere.

Prerequisites

Python 3.9+ — from python.org.

ffmpeg on your PATH (brew install ffmpeg / apt install ffmpeg) — Whisper shells out to it for decoding.

Dependencies:

code

pip install streamlit openai-whisper streamlit-audio-recorder

Step 1: Create the Script

Save as transcriber.py:

code

import streamlit as st
import whisper
import os

st.set_page_config(page_title="Speech to Text", page_icon="🎙️")
st.title("🎙️ Speech to Text — Whisper")

model_size = st.selectbox("Model", ["tiny", "base", "small", "medium"], index=1)
timestamps = st.checkbox("Include timestamps")


@st.cache_resource
def load_model(size):
    return whisper.load_model(size)


uploaded = st.file_uploader("Upload audio", type=["wav", "mp3", "m4a", "ogg", "flac"])

audio_path = None
if uploaded:
    audio_path = f"temp_{uploaded.name}"
    with open(audio_path, "wb") as f:
        f.write(uploaded.getvalue())
else:
    try:
        from audio_recorder_streamlit import audio_recorder
        recorded = audio_recorder(text="Record instead", key="mic")
        if recorded:
            audio_path = "temp_recording.wav"
            with open(audio_path, "wb") as f:
                f.write(recorded)
    except ImportError:
        st.caption("Tip: pip install streamlit-audio-recorder to enable recording.")

if audio_path and st.button("📝 Transcribe", type="primary"):
    model = load_model(model_size)
    with st.spinner(f"Transcribing with {model_size}..."):
        result = model.transcribe(audio_path)

    if timestamps:
        lines = []
        for seg in result["segments"]:
            m, s = divmod(int(seg["start"]), 60)
            lines.append(f"[{m:02d}:{s:02d}] {seg['text'].strip()}")
        transcript = "\n".join(lines)
    else:
        transcript = result["text"].strip()

    st.subheader(f"Transcript ({result.get('language', '?')})")
    st.text_area("Result", transcript, height=300)
    st.download_button("⬇️ Download", transcript, "transcript.txt", "text/plain")
    os.remove(audio_path)

Step 2: Run the App

code

streamlit run transcriber.py

Start with base — it transcribes most clear speech faster than real time on a laptop CPU. Try tiny vs small on an accented clip and compare.

How It Works

Whisper is a sequence-to-sequence transformer trained on 680,000 hours of audio. It works in 30-second windows: audio becomes a spectrogram, and the model decodes text tokens — including punctuation and capitalization, which older engines never handled. Because it was trained on noisy, real-world audio, it degrades gracefully where older tools fail completely.

The model size selector is the app's most honest feature: tiny (39M params) transcribes near-instantly but fumbles names; medium (769M) is markedly better but roughly 10× slower on CPU. Exposing that tradeoff teaches more than picking for the user.

@st.cache_resource keeps the loaded model in memory across reruns — loading small takes ~10 seconds and hundreds of MB, so it must happen exactly once, the same object-caching discipline as the image classifier.

The timestamped mode iterates Whisper's segment list — each segment carries start, end, and text — and formats [mm:ss] cues, the backbone of SRT subtitles if you extend it.

Common Errors & Fixes

`FileNotFoundError: ffmpeg` — Whisper decodes via the ffmpeg CLI; install it and confirm with ffmpeg -version in a *new* terminal.

First transcription is very slow — the model is downloading (tiny ≈ 75MB, medium ≈ 1.5GB); it's cached after the first run.

M4A files fail — old ffmpeg builds lack AAC handling; update ffmpeg.

Out-of-memory on medium — use small, or fp16=False is unnecessary on CPU but needed on some GPU setups with older drivers.

Key Concepts

Local inference — no API keys, no per-minute costs, full privacy.

Model size vs accuracy — an explicit, user-facing tradeoff.

Segment output — timestamps come free with the transcript.

Resource caching — load models once per process, never per rerun.

What to Try Next

Add SRT export — format segments with HH:MM:SS,mmm --> ranges for video subtitles.

Add speaker labels with pyannote-audio diarization on top.

Pipe transcripts into the word cloud generator for meeting summaries.

Translate mode: model.transcribe(path, task="translate") gives English output from any language.

FAQ

How accurate is Whisper?

On clean English audio, small and above rival paid APIs (~5% word error). Accents, crosstalk, and music raise the error — test with your real audio and the timestamped segments to spot trouble.

Does my audio leave my machine?

No — inference is local. The only network traffic is the one-time model download.

Why is CPU transcription slow for long files?

Whisper processes 30-second windows sequentially. tiny runs faster than real time; medium can take 2–3× audio duration. GPU (CUDA) changes the math entirely.