SilentSpeak

Read a word from lip movement alone, live on a webcam, for people the model never saw, without the video ever leaving the device.

LipNet reads whole GRID sentences from pixels with a large network. I went the other way: single words from face landmarks, with a model small enough for a browser tab^{[1][1][1] Credits and related work, in the SilentSpeak README (opens in a new tab)Source 1Credits and related work, in the SilentSpeak READMEOpen source 1 (opens in a new tab)All sources}.

The GRID audio-visual corpus has 33 speakers with video, each reading 1,000 six-word sentences from one fixed grammar (command, colour, preposition, letter, digit, adverb), such as “bin blue at f two now”. Its whole vocabulary is 51 words^{[2][2][2] About the GRID corpus, in the SilentSpeak README (opens in a new tab)Source 2About the GRID corpus, in the SilentSpeak READMEOpen source 2 (opens in a new tab)All sources}.

I split by speaker, so every test number is on faces the model never saw. Six words (set, white, with, k, seven, soon) were also kept out of training entirely, to test the mode where visitors teach their own words^{[3][3][3] Speaker split and segment counts (pipeline/reports/metrics.json) (opens in a new tab)Source 3Speaker split and segment counts (pipeline/reports/metrics.json)Open source 3 (opens in a new tab)All sources}.

• The video stays on the device. Only landmarks reach the model, and nothing is uploaded.
• It has to work on people it never saw, so every number is on held-out speakers.
• A mirrored webcam has to read the same as the training video.
• The Python training features and the TypeScript browser features must match, and tests check it.

MediaPipe face landmarks turned into pose-free, mirror-invariant lip features, and a 1.9 M-parameter convolution and BiGRU encoder trained on the GRID corpus with cross-entropy plus a contrastive loss. It runs in a Web Worker on ONNX Runtime Web, and visitors can teach it their own words from a few takes.

GRID only knows its 51 words, so the open vocabulary comes from teaching: you record three takes each of a few words, and new attempts are matched against them in the model’s embedding space. Three taught words can also drive a small player as a silent remote^{[4][4][4] What you can do, in the SilentSpeak README (opens in a new tab)Source 4What you can do, in the SilentSpeak READMEOpen source 4 (opens in a new tab)All sources}.

Why landmarks, not pixels

The model never sees an image. MediaPipe’s face landmarker turns each frame into 478 3D points, and I keep the 40 on the lips plus the outer eye corners. Coordinates depend far less on lighting, skin tone or camera quality than pixels do, which matters when training video comes from a studio and live video from a webcam. I remove head pose with the face’s transformation matrix and divide sizes by the distance between the eyes, and each left and right pair of lip points is folded into one symmetric feature, so a face and its mirror image give identical input. The cost is that landmarks cannot see the tongue or teeth, so b, p and m look the same^{[5][5][5] Why landmarks, not pixels, in the SilentSpeak README (opens in a new tab)Source 5Why landmarks, not pixels, in the SilentSpeak READMEOpen source 5 (opens in a new tab)All sources}.

The model

Each frame becomes 79 values (the folded lip pairs, the midline points, 16 mouth blendshapes and a face-present flag), and frame-to-frame changes make it 157. Three residual temporal convolutions and a two-layer bidirectional GRU feed an attention pool and a 128-dimensional embedding; the 45 word scores are cosine similarities between that embedding and one learned vector per word. The loss is cross-entropy plus half-weighted supervised contrastive loss, and the contrastive part is what makes the embedding useful for words it never trained on. While you hold the button, frames are resampled to 25 fps and trimmed to where the lips move; a capture that is too short or loses the face is sent back instead of guessed at^{[6][6][6] Features, model and capture, in the SilentSpeak README (opens in a new tab)Source 6Features, model and capture, in the SilentSpeak READMEOpen source 6 (opens in a new tab)All sources}.

It runs on ONNX Runtime Web’s WebAssembly build in a Web Worker. I dropped the WebGPU path: the GRU has no WebGPU kernel and falls back to the CPU anyway, and that runtime adds a 28 MB file. Importing only the WebAssembly build cut the built site from 66 MB to 52 MB^{[7][7][7] The runtime and the dropped WebGPU path, in the SilentSpeak README (opens in a new tab)Source 7The runtime and the dropped WebGPU path, in the SilentSpeak READMEOpen source 7 (opens in a new tab)All sources}.

Checking the browser against training

A quiet mismatch between the Python and TypeScript features would ruin accuracy without any error. Both test suites reproduce the same synthetic fixture to within 1e-4, including a mirrored frame and a resampling case, and a browser test runs MediaPipe on ten real frames of a held-out speaker and compares the result with Python’s. To be sure the check could fail, I read the pose matrix in the wrong layout on purpose: the lip geometry moved by up to 0.012, twelve times the tolerance, and the test catches it^{[8][8][8] Feature parity, in the SilentSpeak README (opens in a new tab)Source 8Feature parity, in the SilentSpeak READMEOpen source 8 (opens in a new tab)All sources}.

Top-1 is the share of words whose first guess is right when the model chooses only among the words of one category, which is how the demo asks: pick a category, then mouth a word from it. Before training I set a bar of 40% top-1 on held-out speakers for a category to ship. For teaching your own words I ran 1,000 rounds of 6-way, 5-shot matching on the six held-out words, with the example takes and the test take from one held-out speaker: a new face, new words and a few takes, close to what a visitor does. Below a threshold calibrated on the validation speakers for 75% precision, the app says “Not sure”^{[9][9][9] How the results are read, the 40% bar and the few-shot test, in the SilentSpeak README (opens in a new tab)Source 9How the results are read, the 40% bar and the few-shot test, in the SilentSpeak READMEOpen source 9 (opens in a new tab)All sources}.

Words never trained on, 6-way, 5 takes each

78.9%

Chance 16.7%

Commands, top-1 on held-out speakers

84.9%

Chance 33.3%

All 45 trained words, top-1

45.9%

Top-5: 80.7%

Model size, ONNX

7.6 MB

1.9 M parameters

Chance is 33.3% in the three-word categories, 11.1% for digits and 4.2% for letters. Across all 45 trained words at once, on 24,316 test segments, top-1 is 45.9% and top-5 is 80.7%. The few-shot result, 78.9% against 16.7% chance, is why teaching your own words is the main mode of the demo^{[10][10][10] Results table (pipeline/reports/metrics.md) (opens in a new tab)Source 10Results table (pipeline/reports/metrics.md)Open source 10 (opens in a new tab)All sources}.

A third of the data was missing

My first full run trained on 14 speakers instead of 25, and nothing crashed. The pipeline paired each alignment folder with a speaker’s video by name, but in the GRID archive 19 of 34 alignment folders carry another speaker’s label. Utterance ids are unique to a speaker, so those folders matched no video and their clips were dropped without an error. The fix matches each speaker’s videos to whichever folder holds the same utterance ids, and fails if no folder covers at least 90% of them. Before trusting the new pairing I checked it against the video: for the remapped speakers, MediaPipe’s jawOpen is 1.6 to 2.6 times higher inside aligned words than in the silences between them. Training segments went from 43,271 to 121,023, and validation top-1 over all words from 0.41 to 0.52^{[11][11][11] What went wrong: a third of the data was silently missing, in the SilentSpeak README (opens in a new tab)Source 11What went wrong: a third of the data was silently missing, in the SilentSpeak READMEOpen source 11 (opens in a new tab)All sources}.

Letters, and other limits

Spelled letters are short and many look alike on the lips, which is why they miss the bar. GRID was filmed in a studio with frontal faces and even lighting, and its faces are few and were recorded in one setup, so other faces, beards, glasses or poor lighting may do worse, and strong head turns break landmark tracking. Even shipped categories are wrong one time in four or five, which is why the headline mode is the one that adapts to your face^{[12][12][12] Model card and limitations, in the SilentSpeak README (opens in a new tab)Source 12Model card and limitations, in the SilentSpeak READMEOpen source 12 (opens in a new tab)All sources}.

The few-shot test uses five takes, but the quick setup in the app asks for three, so I would measure the three-take case directly. And every held-out number is on studio video. The next step is a small set of webcam recordings from volunteers at home, scored next to the GRID numbers, so the gap between the two is a measured number and not a guess.

1. [1]Credits and related work, in the SilentSpeak README (opens in a new tab)Back to where it is cited
2. [2]About the GRID corpus, in the SilentSpeak README (opens in a new tab)Back to where it is cited
3. [3]Speaker split and segment counts (pipeline/reports/metrics.json) (opens in a new tab)Back to where it is cited
4. [4]What you can do, in the SilentSpeak README (opens in a new tab)Back to where it is cited
5. [5]Why landmarks, not pixels, in the SilentSpeak README (opens in a new tab)Back to where it is cited
6. [6]Features, model and capture, in the SilentSpeak README (opens in a new tab)Back to where it is cited
7. [7]The runtime and the dropped WebGPU path, in the SilentSpeak README (opens in a new tab)Back to where it is cited
8. [8]Feature parity, in the SilentSpeak README (opens in a new tab)Back to where it is cited
9. [9]How the results are read, the 40% bar and the few-shot test, in the SilentSpeak README (opens in a new tab)Back to where it is cited
10. [10]Results table (pipeline/reports/metrics.md) (opens in a new tab)Back to where it is cited
11. [11]What went wrong: a third of the data was silently missing, in the SilentSpeak README (opens in a new tab)Back to where it is cited
12. [12]Model card and limitations, in the SilentSpeak README (opens in a new tab)Back to where it is cited

• Python
• PyTorch
• MediaPipe
• ONNX Runtime Web
• TypeScript
• React