GestureSign

Recognize isolated American Sign Language signs from a webcam, for signers the model never saw, on the visitor’s device with no video uploaded.

I trained on asl-signs, Google’s isolated sign recognition dataset on Kaggle, from the PopSign project and licensed CC BY 4.0. It has about 94,000 clips of 250 signs from 21 signers, recorded on phones and supplied as MediaPipe Holistic landmarks rather than video^{[1][1][1] Data, in the GestureSign model card (opens in a new tab)Source 1Data, in the GestureSign model cardOpen source 1 (opens in a new tab)All sources}.

The split is by signer. Three signers are held out for testing and two for validation, so 16 signers are left for training^{[2][2][2] Signer split and clip counts (pipeline/reports/metrics.json) (opens in a new tab)Source 2Signer split and clip counts (pipeline/reports/metrics.json)Open source 2 (opens in a new tab)All sources}.

• No video leaves the page, and the camera starts only after a click.
• Every number is on signers the model never saw.
• Left-handed signers and mirrored cameras must look the same to the model.
• Landmark tracking has to keep up on a mid-range laptop, not only on mine.

MediaPipe Holistic landmarks (hands, upper body and lips), normalized for handedness and camera aspect ratio, and a 1.4 M-parameter convolution and transformer trained on the asl-signs dataset (about 94,000 clips from 21 signers), split by signer. It runs in Web Workers on ONNX Runtime Web and answers “Not sure” below a calibrated confidence.

Choosing the landmarks by measuring first

Before training anything I set a bar: MediaPipe Holistic (hands, pose and face) had to sustain 15 fps in Chrome on my laptop with the CPU throttled 4×, as a stand-in for a mid-range machine. If it missed, the demo would use the separate hand and pose landmarkers and drop the lip features. Holistic ran at a median of 16.8 fps over three runs, so the model sees the hands, the upper body and 40 lip points. The test video was a drawn figure, not a person, so that rate is an upper bound for a real signer^{[3][3][3] Landmark benchmark (pipeline/reports/bench.md) (opens in a new tab)Source 3Landmark benchmark (pipeline/reports/bench.md)Open source 3 (opens in a new tab)All sources}.

What the page does

You sign continuously; a sign ends when your hands leave the frame or hold still for 300 ms, and the page shows its three best guesses. Practice mode picks a sign and scores your attempt. Teach your own records 3 to 5 takes each of up to 8 gestures and matches new attempts to the nearest one in the model’s embedding space, so it works for any gesture, ASL or not. Replay plays held-out clips as landmarks for visitors without a camera^{[4][4][4] What it does, in the GestureSign README (opens in a new tab)Source 4What it does, in the GestureSign READMEOpen source 4 (opens in a new tab)All sources}.

The model

The input is up to 64 frames at 30 fps of 460 landmark features: hands, upper body, lips, presence flags and velocities. Three blocks of depthwise temporal convolution and a transformer feed a masked mean pool and a 128-dimensional embedding, and the 250 sign scores are cosine similarities against it. That is 1,395,520 parameters, 5.6 MB as ONNX. Training used AdamW with warm-up and cosine decay, label smoothing and landmark augmentation (rotation, scale, shear, time stretch, dropped frames and hands), with early stopping on validation top-1; the run stopped after 47 epochs and the checkpoint from epoch 39 ships^{[5][5][5] Architecture and training, in the GestureSign model card (opens in a new tab)Source 5Architecture and training, in the GestureSign model cardOpen source 5 (opens in a new tab)All sources}.

The signing hand always goes in the same slot, and the clip is mirrored if needed. The features exist twice, in Python for training and TypeScript for the browser, and both test suites check one shared fixture to within 1e-4, including mirrored clips, another camera aspect ratio and other frame rates. The page ships only the plain WebAssembly runtime: a model this size runs in a few milliseconds on a CPU, and the WebGPU build would add a 27 MB runtime and need a looser Content Security Policy^{[6][6][6] Design decisions in the GestureSign README (opens in a new tab)Source 6Design decisions in the GestureSign READMEOpen source 6 (opens in a new tab)All sources}.

Top-1 and top-5 are over all 250 signs on the three test signers. One rule was fixed up front: if the full sign set scored under 60% top-1 on held-out signers, the demo would ship the largest subset that reached 60%. The page answers “Not sure” below a confidence threshold calibrated to aim for 80% precision, and teach your own is scored over 1,000 random rounds of 8 gestures with 3 takes each^{[7][7][7] Evaluation settings (pipeline/configs/default.toml) (opens in a new tab)Source 7Evaluation settings (pipeline/configs/default.toml)Open source 7 (opens in a new tab)All sources}.

The full set reached 74.4% top-1 on the validation signers, so all 250 signs ship^{[8][8][8] What ships, in the GestureSign README’s design decisions (opens in a new tab)Source 8What ships, in the GestureSign README’s design decisionsOpen source 8 (opens in a new tab)All sources}.

Top-1 over 250 signs, held-out signers

72.1%

Top-5: 88.1%

Right sign among the 3 shown

85.3%

Teach your own: 8 gestures, 3 takes each

88.7%

Clips answered instead of “Not sure”

92%

77.1% of those answers are right

The page answers 92% of the test clips, and 77.1% of those answers are right, a little under the 80% the threshold aimed for. Teach your own was scored on the test signers’ embeddings too. Most mistakes are near-synonyms or signs made at the same place with a similar hand. Brother begins with the sign for boy, for example. Because the page shows three guesses, the intended sign is usually still on screen when the first guess is its look-alike^{[9][9][9] Results tables (pipeline/reports/metrics.md) (opens in a new tab)Source 9Results tables (pipeline/reports/metrics.md)Open source 9 (opens in a new tab)All sources}.

The portrait and landscape stretch

This one cost me a feature version. MediaPipe divides x by the image width and y by its height. The training videos are portrait phone selfies and webcams are landscape, so on a webcam every vertical distance came out about 1.8× longer than in training. I now scale y by the face’s own proportions (eye to mouth over ear to ear, from pose points), which cancels the camera’s aspect ratio^{[10][10][10] The portrait and landscape stretch, in the GestureSign README’s design decisions (opens in a new tab)Source 10The portrait and landscape stretch, in the GestureSign README’s design decisionsOpen source 10 (opens in a new tab)All sources}.

Limits

Holistic tracks one body, so with several people in view it may follow someone who is not signing. In dim or back-lit rooms MediaPipe loses the hands, and the model only sees what MediaPipe finds. The aspect-ratio fix assumes a typical framing, and facial grammar beyond the lips is not modelled^{[11][11][11] Limitations, in the GestureSign model card (opens in a new tab)Source 11Limitations, in the GestureSign model cardOpen source 11 (opens in a new tab)All sources}.

The landmark benchmark used a drawn figure, so I would run it again with a real person in front of a real webcam before relying on how close Holistic is to the bar. After that, I want accuracy on webcam recordings from signers outside the dataset, since every number here comes from phone selfies like the training clips.

1. [1]Data, in the GestureSign model card (opens in a new tab)Back to where it is cited
2. [2]Signer split and clip counts (pipeline/reports/metrics.json) (opens in a new tab)Back to where it is cited
3. [3]Landmark benchmark (pipeline/reports/bench.md) (opens in a new tab)Back to where it is cited
4. [4]What it does, in the GestureSign README (opens in a new tab)Back to where it is cited
5. [5]Architecture and training, in the GestureSign model card (opens in a new tab)Back to where it is cited
6. [6]Design decisions in the GestureSign README (opens in a new tab)Back to where it is cited
7. [7]Evaluation settings (pipeline/configs/default.toml) (opens in a new tab)Back to where it is cited
8. [8]What ships, in the GestureSign README’s design decisions (opens in a new tab)Back to where it is cited
9. [9]Results tables (pipeline/reports/metrics.md) (opens in a new tab)Back to where it is cited
10. [10]The portrait and landscape stretch, in the GestureSign README’s design decisions (opens in a new tab)Back to where it is cited
11. [11]Limitations, in the GestureSign model card (opens in a new tab)Back to where it is cited

• Python
• PyTorch
• MediaPipe
• ONNX Runtime Web
• TypeScript
• React