Images, language and games
Machine learning for OCR
Machine learning for OCR finds text regions in scans and photos, then reads each word or line with a recognition model, including handwriting. Layout analysis keeps reading order, columns and tables intact, and post-OCR correction fixes likely errors before the text is searched or extracted.
How machine learning is used in OCR
Recognising printed and handwritten text
Recognition models read a cropped line or word image and output the characters, usually with an image encoder and a text decoder. Modern models handle many fonts and handwriting styles.
Real exampleTrOCR (Microsoft)TrOCR is a transformer-based OCR model that uses pretrained image and text transformers for printed and handwritten text recognition.
Transcribing historical handwriting
Archives train recognition models on transcribed pages of a particular script or collection, then apply them to the rest. This makes centuries of handwritten records searchable.
Real exampleTranskribusTranskribus is a platform for recognising handwritten and printed text in historical documents, run by a cooperative and offering community-trained AI models.
Document layout analysis
Before reading text, a model finds titles, paragraphs, tables, figures and captions on each page. This keeps reading order and table structure intact when documents are converted.
Real exampleDocLayNet (IBM)DocLayNet is a public dataset of 80,863 manually annotated pages from diverse sources for training document layout models.
Understanding forms and receipts
Models that combine the text with its position on the page can link labels to values, such as an invoice total or a form field. This turns scanned forms into structured records.
Real exampleLayoutLM (Microsoft)LayoutLM jointly models text and layout information from scanned documents and was evaluated on form understanding, receipt understanding and document image classification.
Can you learn machine learning for OCR?
Yes. You can get useful OCR results with existing engines and a little Python, and then learn to fine-tune recognition and layout models. Learn Python and image handling first, then evaluation with character error rate, then deep learning for sequence recognition.
- Python and image preparationLoad scans, deskew, crop and binarise pages with OpenCV before recognition.Learn it in Python
- Measuring OCR qualityCompute character and word error rates against ground-truth transcriptions, and look at error types by hand.
- Neural networks for sequencesLearn CNN and transformer encoders and the decoders that turn image features into text.Learn it in Deep Learning
- Fine-tuning on your documentsTranscribe a few hundred lines from your own collection and fine-tune a recognition model such as TrOCR or docTR.
Moving into OCR and document AI
If you already work in the field
If you work in archives, records management or document processing, you know the scripts, layouts and error costs. Learn Python and an OCR toolkit, then measure error rates on a sample of your own pages before and after fine-tuning.
If you come from data science or ML
If you come from machine learning, learn how scanning, fonts, scripts and page layout affect recognition. Spend time with the people who use the text, because acceptable error rates depend heavily on the task.
Roles that do this workDocument AI engineerOCR engineerMachine learning engineer, document processingDigital archivist
Tools you will use
| Tool | What you use it for |
|---|---|
| Tesseract | Open-source OCR engine with many trained languages, the usual baseline for printed text. |
| docTR | Deep-learning text detection and recognition for documents, returning words with their positions. |
| PaddleOCR | Multilingual text detection, recognition and document layout analysis with ready-made pipelines. |
| TrOCR | Transformer OCR model in Hugging Face, with checkpoints for printed and handwritten text lines. |
| OpenCV | Deskew, threshold, denoise and crop page images before recognition. |
| jiwer | Compute character and word error rate between OCR output and reference transcriptions. |
Public data to practise on
- FUNSD: Scanned forms with word boxes, transcriptions and question-answer links, for OCR and form understanding.
- CORD: Receipt images with text and labelled fields, for OCR and post-OCR parsing of items and totals.
- Bentham Dataset R0: Handwritten manuscripts by Jeremy Bentham with layout and line-level transcriptions, for handwriting recognition.
Your first project
Download FUNSD, crop each annotated word, recognise it with Tesseract as a baseline and with docTR's recognition model, and compare their character and word error rates using jiwer.
Common questions
Is OCR machine learning?
Modern OCR engines use machine learning for both text detection and recognition. Older systems relied on hand-built rules and template matching.
Can OCR read handwriting?
Yes, handwritten text recognition models can, especially after fine-tuning on samples of the same hand or script. Accuracy varies widely with writing quality and training data.
Is Tesseract still good enough?
Tesseract works well on clean printed text and is a good free baseline. Deep learning engines usually do better on photos, complex layouts and handwriting.
Courses for these skills
- Deep Learning with PyTorchLearn image representations and reconstruction objectives.Syllabus available · lessons open as produced
- Generative AI: LLMs, RAG and AgentsUse structured-output checks when building a later document assistant.Syllabus available · lessons open as produced
- Data Tools: NumPy, pandas, Matplotlib and scikit-learnKeep annotation coordinates and image identifiers aligned.23 published lessons · all planned lessons open
- Maths for Machine LearningConnect losses and gradients to pixel-level changes.26 published lessons · all planned lessons open
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.