Speech & Audio
For ownersCalls turned into text and text turned into a voice: the base of an AI phone line.
The other half of human communication. Hear, understand, speak.
The lessons
- 01Audio Fundamentals — Waveforms, Sampling, Fourier TransformSound in a computer is a long list of loudness readings taken thousands of times per second. The Fourier transform breaks that signal into the frequencies it contains.Learn · ~45 minutes
- 02Spectrograms, Mel Scale & Audio FeaturesA spectrogram is a picture of sound showing which frequencies are loud at each moment. Mel spectrograms stretch that picture to match how human hearing works, which suits AI models.Build · ~45 minutes
- 03Audio Classification — From k-NN on MFCCs to AST and BEATsAudio classification gives a sound clip a label, such as alarm, speech, music or a specific language. Methods range from simple feature matching to transformer models trained on huge sound libraries.Build · ~75 minutes
- 04Speech Recognition (ASR) — CTC, RNN-T, AttentionSpeech recognition turns spoken audio into text. Models use methods called CTC, RNN-T or attention to line up sounds with letters and words over time.Build · ~45 minutes
- 05Whisper — Architecture & Fine-TuningWhisper is an open speech recognition model from OpenAI. It reads audio in short windows and outputs text, handling many languages, translation and timestamps in one model.Build · ~75 minutes
- 06Speaker Recognition & VerificationSpeaker recognition works out who is speaking rather than what they said. It turns a voice into a fingerprint of numbers and compares it with stored voice fingerprints.Build · ~45 minutes
- 07Text-to-Speech (TTS) — From Tacotron to F5 and KokoroText-to-speech turns written text into spoken audio. Modern systems convert text into sound units, predict a spectrogram, then use a vocoder to produce a natural-sounding waveform.Build · ~75 minutes
- 08Voice Cloning & Voice ConversionVoice cloning makes a model speak any text in a chosen person's voice. Voice conversion changes a recording so it sounds like someone else while keeping the words.Build · ~75 minutes
- 09Music Generation — MusicGen, Stable Audio, Suno, and the Licensing EarthquakeMusic generation models create songs or instrumental tracks from text prompts, melodies or reference audio. Commercial and open models now produce usable results, but licensing rules are still unsettled.Build · ~75 minutes
- 10Audio-Language Models — Qwen2.5-Omni, Audio Flamingo, GPT-4o AudioAudio-language models listen to speech, sounds and music and answer questions about them in text. They connect an audio encoder to a large language model.Learn · ~45 minutes
- 11Real-Time Audio ProcessingReal-time audio processing handles sound in tiny chunks as it arrives, fast enough that people do not notice a delay. It is how live calls and voice agents stay responsive.Build · ~75 minutes
- 12Build a Voice Assistant Pipeline — The Phase 6 CapstoneA voice assistant chains several parts: detect speech, transcribe it, let a language model reply, then speak the answer. This capstone builds that loop end to end.Build · ~120 minutes
- 13Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC and the Semantic-Acoustic SplitNeural audio codecs compress sound into a short sequence of tokens and rebuild it later. These tokens let language-style models generate and understand audio the way they handle text.Learn · ~60 minutes
- 14Voice Activity Detection & Turn-Taking — Silero, Cobra, and the Flush TrickVoice activity detection spots when someone is speaking. Turn-taking logic decides when they have finished, so a voice agent knows when to reply and when to stop talking.Build · ~45 minutes
- 15Streaming Speech-to-Speech — Moshi, Hibiki, and Full-Duplex DialogueStreaming speech-to-speech models listen and talk at the same time in one model, instead of chaining transcription, a language model and voice output. Moshi and Hibiki are open examples.Learn · ~75 minutes
- 16Voice Anti-Spoofing & Audio Watermarking — ASVspoof 5, AudioSeal, WaveVerifyAnti-spoofing models try to tell real human speech from AI-generated or replayed voices. Audio watermarking hides an inaudible mark in generated audio so it can be identified later.Build · ~75 minutes
- 17Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open LeaderboardsAudio evaluation uses standard scores to judge systems: word error rate for transcription, listener ratings for voice quality, and benchmarks for audio understanding and generation.Learn · ~60 minutes
Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License
Next step
Rather have it built for you?
Reading is optional. Tell me what's stuck and I'll build it into the business you already run.