Speech-to-Text and Text-to-Speech with Modern AI

Voice used to be a specialist feature. Now a laptop can transcribe speech offline, an API can read text aloud in a natural voice, and assistants can hold spoken conversations. The building blocks are speech-to-text, which turns audio into words, and text-to-speech, which does the reverse.

This guide explains how each works, measures two open speech recognition models, and covers the choices that matter in an application: accuracy, cost, privacy, licences and disclosure.

How speech-to-text works

Modern recognisers are trained on very large amounts of transcribed audio. Whisper, the best-known open family, was trained on 680,000 hours of multilingual and multitask data, and generalises to new datasets without fine-tuning (Radford et al.: Whisper).

The pipeline is easy to picture. Audio is resampled to 16,000 samples a second and turned into a spectrogram: 80 frequency bands measured every 10 milliseconds, an image of the sound. An encoder reads that image in 30-second windows, and a decoder writes the transcript token by token, much as a language model writes text. See how LLMs work.

Whisper comes in five sizes, and the smallest four have English-only versions (Hugging Face: whisper-tiny.en).

Two models, 30 clips

The script transcribes 30 clips of read English from LibriSpeech, a corpus of about 1,000 hours of audiobook speech under CC BY 4.0 (OpenSLR: LibriSpeech). It compares the two smallest English-only Whisper models by word error rate (WER): the number of words substituted, inserted or deleted, divided by the words in the reference.

import io
import re
import time

import pandas as pd
import soundfile as sf
from transformers import pipeline

# 30 clips of read English speech from LibriSpeech (CC BY 4.0), with their reference transcripts
clips = pd.read_parquet("librispeech_dummy.parquet").head(30)
audio = [sf.read(io.BytesIO(a["bytes"])) for a in clips.audio]
seconds = sum(len(wave) / rate for wave, rate in audio)
print(f"{len(clips)} clips, {seconds / 60:.1f} minutes of speech")


def words(text):
    return re.sub(r"[^a-z' ]", " ", text.lower()).split()


SPELLINGS = {"mr": "mister", "mrs": "missus", "dr": "doctor"}    # LibriSpeech spells these out


def normalised(text):
    return [SPELLINGS.get(w, w) for w in words(text)]


def word_errors(ref, hyp):
    # edit distance over words: substitutions, insertions and deletions
    d = list(range(len(hyp) + 1))
    for i, r in enumerate(ref, 1):
        prev, d[0] = d[0], i
        for j, h in enumerate(hyp, 1):
            prev, d[j] = d[j], min(d[j] + 1, d[j - 1] + 1, prev + (r != h))
    return d[-1]


results = {}
for name in ("openai/whisper-tiny.en", "openai/whisper-base.en"):
    asr = pipeline("automatic-speech-recognition", model=name)
    start = time.perf_counter()
    texts = [asr({"raw": wave, "sampling_rate": rate})["text"] for wave, rate in audio]
    elapsed = time.perf_counter() - start
    total = sum(len(words(r)) for r in clips.text)
    raw = sum(word_errors(words(r), words(h)) for r, h in zip(clips.text, texts))
    fair = sum(word_errors(normalised(r), normalised(h)) for r, h in zip(clips.text, texts))
    results[name] = texts
    print(f"\n{name}: {asr.model.num_parameters() / 1e6:.0f}M parameters")
    print(f"   word error rate as written: {raw / total:.1%} ({raw} errors in {total} words)")
    print(f"   after normalising spellings such as mr and mister: {fair / total:.1%} ({fair} errors)")
    print(f"   speed on this laptop: {seconds / elapsed:.1f} times real time")

print("\nwhere they differ, first three clips:")
shown = 0
for ref, tiny, base in zip(clips.text, *results.values()):
    if words(tiny) != words(base) and shown < 3:
        shown += 1
        print(f"reference: {ref.lower()}")
        print(f"tiny:      {' '.join(words(tiny))}")
        print(f"base:      {' '.join(words(base))}\n")

30 clips, 3.8 minutes of speech

openai/whisper-tiny.en: 38M parameters
   word error rate as written: 8.0% (44 errors in 549 words)
   after normalising spellings such as mr and mister: 6.6% (36 errors)
   speed on this laptop: 1.6 times real time

openai/whisper-base.en: 73M parameters
   word error rate as written: 9.1% (50 errors in 549 words)
   after normalising spellings such as mr and mister: 7.7% (42 errors)
   speed on this laptop: 1.0 times real time

where they differ, first three clips:
reference: he has grave doubts whether sir frederick leighton's work is really greek after all and can discover in it but little of rocky ithaca
tiny:      he has grave doubts whether sir frederick layton's work is really greek after all and can discover in it but little of rocky ithaca
base:      he has grieved doubts whether sir frederick layton's work is really greek after all and can discover in it but little of rocky ithaca

reference: linnell's pictures are a sort of up guards and at em paintings and mason's exquisite idylls are as national as a jingo poem mister birket foster's landscapes smile at one much in the same way that mister carker used to flash his teeth and mister john collier gives his sitter a cheerful slap on the back before he says like a shampooer in a turkish bath next man
tiny:      linnell's pictures are a sort of upguards and atom paintings and mason's exquisite idles are as national as a jingo poem mr birkett foster's landscapes smile at one much in the same way that mr carker used to flash his teeth and mr john collier gives his sitter a cheerful slap in the back before he says like a shampooer and a turkish bath next man
base:      lynille's pictures are a sort of up guards and atom paintings and mason's exquisite idols are as national as a jingo poem mr burkett foster's landscapes smile at one much in the same way that mr carker used to flash his teeth and mr john collier gives his sitter a cheerful slap on the back before he says like a shampoo or a turkish bath next man

reference: on the general principles of art mister quilter writes with equal lucidity
tiny:      on the general principles of art mr quilter writes with equal lucidity
base:      under general principles of art mr quilter writes with equal lucidity

What the output shows

• Both models are usable on a laptop. Each made fewer than one error in ten words, and kept pace with the audio on a laptop CPU that was busy with other work: tiny at 1.6 times real time, base at about real time. Speed varies with the machine and its load.
• Normalising spellings removed eight errors from each. LibriSpeech writes "mister" where Whisper writes "Mr". These are style differences, not mistakes. The Whisper authors make the same point: WER penalises innocuous differences in transcript style, so they built a text normaliser before scoring.
• The larger model did not win here. Base made 42 errors after normalisation to tiny's 36. On 549 words that is a difference of six errors, too small to rank the models. On a larger test, or on your own audio, the result could reverse.
• The real errors are names and rare words. "Leighton" became "Layton", "idylls" became "idles" and "idols". Proper nouns and unusual vocabulary are where any recogniser struggles, and where a list of expected terms helps most.

The practical lesson is about measurement. Compare models with the same normalisation, on audio like yours, with enough words to make a difference meaningful.

Local model or API

Hosted options change quickly. On 3 October 2026, OpenAI's transcription guide recommended gpt-transcribe for recorded speech, with files up to 25 MB (OpenAI: speech to text). Its deprecations page lists whisper-1 and the GPT-4o transcription models for shutdown on 26 February 2027 (OpenAI: deprecations). The open Whisper models used above are separate downloads under their own licence and are not affected.

How text-to-speech works

Speech synthesis runs the other way: text is turned into a representation of sound, then into an audio waveform. Neural models now produce voices with natural rhythm and emphasis, and many accept instructions about tone.

OpenAI's text-to-speech guide on 3 October 2026 described a GPT-4o mini TTS model with 11 built-in voices. It also stated that OpenAI's usage policies require telling listeners that the voice is AI-generated (OpenAI: text to speech). The deprecations page, updated on 1 October 2026, lists tts-1, tts-1-hd and the GPT-4o mini TTS snapshots for shutdown on 6 January 2027, with a realtime model as the replacement. Check both pages before building on a specific model.

Open voice models exist too, and their licences vary widely. Some popular ones permit only non-commercial use. Read the licence before a voice ships in a product.

Building a voice assistant

The classic pipeline chains three models: speech-to-text, a language model, then text-to-speech. It is easy to build and debug, but each hop adds delay. Speech-to-speech models handle audio directly and respond faster. See multimodal AI.

Whatever the design, measure the transcription step on your users' audio. Accents, background noise, phone lines and domain vocabulary all change error rates.

Common mistakes

• Comparing WER figures computed with different normalisation.
• Testing on clean audiobooks and deploying on noisy phone calls.
• Ignoring names and jargon, where most real errors are.
• Not telling users a voice is synthetic.
• Shipping a voice model whose licence forbids commercial use.
• Building on a hosted model without checking its deprecation date.

Multimodal models that work with text, images and audio are covered in the Multimodal module of the Generative AI course.

Sources

• Robust Speech Recognition via Large-Scale Weak Supervision (arXiv) — Radford et al.
• LibriSpeech ASR corpus (OpenSLR) — OpenSLR
• whisper-tiny.en model card (Hugging Face) — OpenAI
• File transcription (OpenAI API documentation) — OpenAI
• Text to speech (OpenAI API documentation) — OpenAI
• Deprecations (OpenAI API documentation) — OpenAI