Background: For years, I had a habit of writing whenever I traveled. No laptop, no apps, just a pen, a notebook, and pages of cursive observations about airports, flights, fellow travelers, and the emotions that come with being away from home. Two decades later, I decided to digitize those notebooks and make them available on Kindle, eventually publishing them as Transit Tales: Between Airports and Emotions: Travel Blog 20 Years Too Late (View on Amazon). As a first step, I assumed modern LLM-powered handwriting recognition would effortlessly convert the pages into text. Instead, the journey led to an unexpected discovery: handwriting recognition is still far from "solved."
The apparent success of modern handwriting recognition is increasingly a combination of imperfect OCR and powerful language-model reconstruction. What users see is often not what the system read, but what the system inferred.
Handwriting Recognition: Better Than It Looks, But Not For The Reason You Think
Modern handwriting recognition demos can be surprisingly impressive. Feed a handwritten page such as the one below into a state-of-the-art OCR system and the resulting transcript may appear almost perfect at first glance. However, a closer examination reveals that the raw recognition quality is often much poorer than it seems. The magic comes not from character recognition alone, but from the language model sitting behind it.
Consider several examples from this handwritten page. The handwritten phrase corresponding to"airport" may initially be recognized as something closer to"surpat". Similarly,"vehicle" can become"valuele","takes a U-turn" becomes"tales a Utum","X-ray machine" becomes"A-vay machine", and"immigration card" appears as"immerdian card". Taken literally, these outputs are poor transcriptions and would be difficult for a human reader to understand without context.
Large language models change the game. Given the surrounding sentence, the model realizes that a traveler arriving at a terminal is far more likely to write"airport" than"surpat","vehicle" than"valuele", and"X-ray machine" than"A-vay machine". The model effectively performs error correction using context, world knowledge, and statistical expectations about language. In other words, it is not necessarily reading the handwriting correctly; it is often inferring what the author probably meant.
This distinction becomes even more obvious with longer phrases. The raw transcription contains fragments such as"want continues - the grove length increase" which are difficult to interpret. Yet an LLM can confidently reconstruct the phrase as"wait continues - the queue length increases". Likewise,"dot Mupose" can be corrected to"what purpose","quese" to"queue", and"lanton" to"London". These corrections are plausible and often accurate, but they are driven at least as much by language understanding as by handwriting recognition itself.
There is an important lesson here. Traditional OCR performance metrics focus on recognition accuracy: did the system correctly identify the handwritten characters? Large language models introduce a second layer that asks a different question: given a noisy transcription, what is the most likely intended text? For users, the final output looks dramatically better, creating the impression that handwriting recognition has become nearly solved. In reality, the underlying recognizer may still be making substantial mistakes, many of which are quietly repaired by contextual reasoning.
The result is both impressive and potentially misleading. If the document contains common travel terms such as airport, boarding pass, London, or immigration card, contextual correction works remarkably well. But when the document contains uncommon names, technical terminology, passwords, identifiers, code snippets, or domain-specific jargon, the same mechanism can "correct" the text into something that is fluent but wrong. Put differently, today's systems often excel not because they perfectly read handwriting, but because they are extraordinarily good at guessing what the handwriting was intended to say.
Summary: Looking at the sample page below, the raw transcription produced words such as"surpat" for airport,"valuele" for vehicle,"A-vay machine" for X-ray machine, and "immerdian card" for immigration card. Yet the final output often looks remarkably accurate because the language model uses context to infer what was probably intended. In many cases, what appears to be excellent handwriting recognition is actually a combination of imperfect character recognition and powerful language understanding. The real breakthrough is not that the machine can read every handwritten word, but that it can often guess the right one.
Extra Reading:
Confidence is the LLM estimate of how likely the "best guess" matches the intended handwritten word.
| Handwritten text | Best guess | Confidence |
|---|
| bleam | because / blame | 20% |
| surpat | airport | 90% |
| valuele | vehicle | 95% |
| cann | came | 85% |
| tales a Utum | takes a U turn | 95% |
| 1000per | 10:00 pm | 90% |
| fright | flight | 99% |
| instid enquiry | initial enquiry | 80% |
| want continues | wait continues | 95% |
| grove length | queue length | 90% |
| tweekning | screening | 60% |
| whoum | hmm / whomm | 30% |
| bequin | begin | 95% |
| mere slowly | move slowly | 95% |
| A-vay machine | X‑ray machine | 98% |
| boarding fan the west item | boarding pass, the next item | 35% |
| had baggage | hand baggage | 98% |
| lanton | London | 99% |
| yh | yes | 95% |
| dot Mupose | what purpose | 90% |
| fan et arom | from her arm / from a folder | 20% |
| gilt the boarding tam | gives the boarding pass | 85% |
| should Day are of the ben pleasant girls | should say one of the best pleasant girls | 70% |
| tack your bags | lock your bags | 75% |
| 20 1 | TSA lock? / combination lock? | 10% |
| immerdian card | immigration card | 99% |
| quese | queue | 99% |
| sigafare vise page | Singapore visa page | 80% |
| Monty me in 3001 | cost me Rs 300/- | 65% |
| hundley | hundred | 85% |
Places where context helps.
| Original handwriting | Most likely intended text |
|---|
| “the valuele in which I cann goes all the way to the end are tales a Utum” | “the vehicle in which I came goes all the way to the end and takes a U turn” |
| “want continues - the grove length increase” | “wait continues - the queue length increases” |
| “both checked in and have baggage go through the A-vay machine” | “both checked-in and hand baggage go through the X-ray machine” |
| “for dot Mupose” | “for what purpose” |
| “Fill the immerdian card” | “Fill the immigration card” |
| “as my sigafare vise page” | “on my Singapore visa page” |
| “Monty me in 3001” | “cost me Rs. 300/-” |
