How to Make Handwritten Notes and Sketches Actually Searchable
""What Apple, Google, and Microsoft already built — and the real gap none of them cover""
Say you've got a shelf of notebooks, a drawer of index cards, and a phone full of photos of whiteboards from old work meetings. Somewhere in that pile is the answer to something you need right now — which notebook has the electrician's number, which sketch has the parts list for the bike you're fixing, which whiteboard photo has the budget figure someone wrote down and never typed up. A typed document you can search in two seconds. Handwriting is a genuinely different problem, and it's one that three major tech companies have already put real engineering into, with three different results.
People searching "how to search handwritten notes" or "OCR app for handwriting" are usually trying to solve one of two different problems, and it's worth separating them. One is transcription: turning ink into typed, searchable text, which is what Apple's Notes app, Google Photos, and Microsoft's OneNote already do, each in a different way and with different trade-offs around where the processing happens and what counts as supported handwriting. The other is comprehension: asking an actual question about what's in an image — including a sketch that has no legible words in it at all — and getting an answer, not just a match.
This piece goes through what those three tools actually do today, with sources, including where each one still falls short. Then it gets into the second problem, the one none of them solve: asking a real question about an image or a mixed folder of scanned notes, PDFs, and hand-drawn sketches, entirely offline.
What Apple, Google, and Microsoft Already Do With Handwriting
Three products already handle a version of this, and the differences between them matter more than most people realize. One works entirely on the device as you write. One works over your whole cloud photo library. One only works if you've turned on a specific cloud sync setting. Here's what each one is actually doing, not just what it's marketed as doing.
Apple's approach lives inside Notes on iPad. When you write with an Apple Pencil, a feature called Scribble converts the strokes to text in the background as you write — there's no separate scan step. Apple's own support documentation describes on-device tools like spellcheck and handwriting auto-refine that run on top of that converted text, correcting misspelled words and adjusting letterforms while keeping the note looking handwritten. Apple's search guide for the app confirms what that background conversion is actually for: the same search box that covers typed text, text in images, and scanned documents also covers handwritten pages — type a word into Notes search and it can surface a page that was never manually transcribed. The catch is language support: Scribble launched with iPadOS 14 in 2020 with English and Chinese only, and French, German, Italian, Portuguese, and Spanish weren't added until the iPadOS 14.5 update the following year, according to MacRumors' reporting at the time. If your iPad's system language isn't on that list, the search benefit doesn't apply.
Google's version lives in Google Photos rather than a notes app, and it works on any text in a photo, typed or handwritten, wherever it appears. Per Google's own help page, wrapping a search term in quotation marks gives an exact text match against filenames, captions, and text found within the photo; without quotes, results widen beyond exact text. It's genuinely useful for finding one photographed page inside a library of thousands of pictures, but it's search over a flat photo library — it points you to the photo, it doesn't read or answer a question about what's written in it.
Microsoft's version sits inside OneNote, and Microsoft has been the most specific about how it actually works. Ink from a stylus is uploaded and run through Microsoft's recognition engine, and per Microsoft's own announcement of the feature, which still describes the underlying mechanism, that processing takes two to five minutes and only applies to notebooks saved to OneDrive — a notebook kept purely offline isn't covered. The same post puts the recognizer's language support at 25 languages. Microsoft also makes a separate scanning app, Microsoft Lens, which according to its own support page is best used for capturing handwritten notes and sketches on a dry erase board, and can turn a photographed page into an editable Word file using OCR. That's a different job than OneNote's ink search: it's OCR run on a photo rather than recognition of live pen strokes, and like any photo-based OCR, results generally hold up better on a clean, well-lit whiteboard shot than on handwriting photographed at an angle or in dim light.
What none of the three actually do is answer a question about an image. They convert handwriting into text you can then search for, or copy text out of a photo you already found. None of them will look at a hand-drawn sketch with no legible words in it and tell you what's drawn.
- Cursive handwriting is still the hard case for every recognition engine on this list, not just the smaller ones. An independent comparison of desktop OCR tools by imagetotable.ai found ABBYY FineReader's accuracy dropping from 95.2% on printed handwriting to 91.7% on cursive, and Adobe Acrobat dropping further, from 88.6% to 79.3% — roughly a 10 to 15 percentage-point penalty for joined-up letters, because cursive removes the clear gaps between characters that recognition depends on.
- Language coverage is uneven and easy to miss, and it can also depend on which device wrote the ink in the first place. Scribble's non-English languages arrived a full iPadOS version behind English, and OneNote's ink recognizer covers 25 languages — a lot, but not universal. A Microsoft support thread confirms OneNote for iPad doesn't have the ink-to-text feature at all — handwriting written there only becomes searchable once the same notebook is opened in OneNote for Windows, where the recognizer actually runs.
- A folder that mixes formats is the case none of the three tools above is built for. Say a shelf of board games has PDF rulebooks plus an index card where you scribbled a house rule mid-game — LocalDocs can be pointed at that whole folder and asked a rules question that draws on both the printed manual and the handwritten card, and it names which document the answer came from.
- LocalDocs runs its recognition model on the phone itself, not a data-center GPU, and that trade-off shows up as speed: indexing a large folder, or asking a question that has to check many documents at once, takes noticeably longer than sending the same request to a big cloud model. The upside is that nothing has to leave the phone to get an answer.
- Handwriting understanding on LocalDocs, like every tool above, is strongest on legible print or a clearly drawn sketch, and weaker on fast cursive or a diagram that crowds sketches and scrawled labels together. Neater notes get meaningfully better results than rushed ones.
LocalDocs is built for exactly that mixed-folder case. Point it at a folder — the app's own promotional materials use an example of 23 documents across 10 folders — and it reads PDFs, Word files, plain text, Markdown, HTML pages, photos, and hand-drawn sketches together, then answers a plain-language question with a citation showing which specific document it pulled from; tap the citation and it opens the source so you can check the answer yourself instead of taking it on faith. Answers can be copied, shared, or read aloud, and question history is grouped by day with a source count next to each answer. It runs on an Android 12 or later, 64-bit device, entirely on the phone: no account, no analytics, no tracking, and no internet connection needed once a folder is indexed. That's a narrower claim than "private," and a more precise one — it means documents and questions are processed on the device and are not uploaded to a server to generate the answer. It doesn't mean the phone itself is secure: anyone who picks up an unlocked phone can open the app, and on-device processing does nothing to stop malware already on that phone. It also doesn't retroactively protect files that are already synced elsewhere — if the folder LocalDocs is pointed at sits inside Google Drive or a similar sync service, those files are already in the cloud regardless of how LocalDocs itself handles them. And because the model is small enough to run on a phone, it can still get an answer wrong the way any language model can; the tappable citation exists so you can check it rather than trust it outright.
LocalDocs is Mystum's own app — we built it, and we're telling you about it here directly. If you want to try pointing it at your own folder of scanned notes, sketches, and documents, you can get it here.
The Actual Takeaway
If you're already inside one of these ecosystems, use what's already there before reaching for anything new. An iPad and Apple Pencil already make your own handwriting searchable as you write it, no extra app required. A OneNote notebook synced to OneDrive already turns your ink searchable within a few minutes. A Google Photos library already lets you search text inside any photo you've taken, handwritten or not. All three are free, all three are already built, and none of them require you to change how you take notes.
Where they run out of road is the same place: none of them can look at a sketch and tell you what it shows, and none of them can answer a question that spans a folder of mixed formats — a scanned manual, a typed document, and a handwritten card — in one pass. That's a narrower job, and it's the one LocalDocs is built around, with the trade-offs that come with running a language model on a phone instead of a server: slower on large folders, and, like the built-in tools above, still weaker on messy cursive than on clear print.
None of this makes handwriting recognition a solved problem. It makes it a set of tools that are each good at a specific, narrower thing than the marketing around them suggests — and picking the right one starts with being honest about which of those narrower things you actually need.