Shashank Yadav
Resume
Back to work

Case study

AccuScan

A handwriting reader in the browser: a 4.8 MB model reads a written word or a photographed line on your device.

Code on GitHub (opens in a new tab)Live demo (opens in a new tab)20.1% character error rate on held-out GNHK words (TrOCR-small: 51.4%)
01

Problem

Read handwriting on the visitor’s own device, with a model small enough to download into a browser tab, measured on photographed notes it never saw.

02

Constraints and approach

  • Small enough to download into a browser tab, and fast on one CPU thread.
  • Nothing uploaded: drawings and photos are read on the device.
  • Measured on held-out photographed handwriting, against a public baseline.

A CNN and BiLSTM with CTC, trained on GNHK and 150,000 synthetic lines in 54 handwriting fonts, quantized to int8 ONNX and run in a Web Worker. Preprocessing and decoding are written in both Python and TypeScript and checked against the same fixtures, so the browser feeds the model exactly what it trained on.

I trained it on my laptop’s GPU. Partway through, the memory cache grew past the machine’s 16 GB and epochs slowed from about 6 minutes to 33; I fixed it, resumed from the saved state without losing work, and finished the run[1]Source 1Training run and what went wrong, in the AccuScan READMEOpen source 1 (opens in a new tab)All sources.

03

Results

The test set is GNHK’s official test split: photographed handwriting from many writers that the model never saw. Lower error rates are better[2]Source 2Results tables (pipeline/reports/metrics.md)Open source 2 (opens in a new tab)All sources.

Character error rate, GNHK test words
20.1%
TrOCR-small zero-shot: 51.4%
Character error rate, GNHK test lines
21.5%
TrOCR-small zero-shot: 38.8%
GNHK test words read exactly
56.8%
64.3% with the dictionary
Model size, int8 ONNX
4.8 MB
TrOCR-small: 246 MB
CPU time for one word, single thread
6 ms
TrOCR-small: 80 ms
Character error rate on GNHK testLower is better.
Character error rate of AccuScan and TrOCR-small on GNHK test words and lines, as percentages
SetupCharacter error rate
AccuScan, words
20.1%
TrOCR-small, words
51.4%
AccuScan, lines
21.5%
TrOCR-small, lines
38.8%

Size and speed

AccuScan and TrOCR-small compared
MeasureAccuScanTrOCR-small
Size4.8 MB (int8)246 MB
CPU time per word6 ms80 ms
Parameters4.0 million62 million

The README explains the comparison and its limits in full[3]Source 3How to read the TrOCR comparison, in the AccuScan READMEOpen source 3 (opens in a new tab)All sources.

04

Starting over

My first version was a notebook that fine-tuned a Donut model. It was never run end to end, so its accuracy figures could not be reproduced, and it had a training bug that kept the model from learning where to stop. I rebuilt it as a model sized for the browser, trained on data whose licences allow publishing the weights, and evaluated it with standard character and word error rates[4]Source 4v1 to v2, in the AccuScan READMEOpen source 4 (opens in a new tab)All sources.

05

Sources

  1. [1]Training run and what went wrong, in the AccuScan README (opens in a new tab)Back to where it is cited
  2. [2]Results tables (pipeline/reports/metrics.md) (opens in a new tab)Back to where it is cited
  3. [3]How to read the TrOCR comparison, in the AccuScan README (opens in a new tab)Back to where it is cited
  4. [4]v1 to v2, in the AccuScan README (opens in a new tab)Back to where it is cited