Business and operations
Machine learning for libraries
Public, academic and national libraries use machine learning to suggest subject headings for new records, enrich and clean catalogue metadata, and power search and recommendations across collections. Text classification and embeddings help cataloguers work faster and readers find related items.
How libraries use machine learning
Suggesting subject headings
Subject indexing models learn from records that cataloguers have already described and suggest terms from a controlled vocabulary for new items. Cataloguers review the suggestions, which speeds up work on large backlogs. Text classifiers and ensembles of lexical and statistical methods are used.
Real exampleAnnif (National Library of Finland)Annif is an open-source tool for automated subject indexing led by the National Library of Finland and used by organisations including the German National Library and the National Library of Sweden.
Finding duplicate catalogue records
Shared catalogues collect many records for the same item from different libraries. A model compares record pairs on titles, authors, dates and identifiers and predicts whether they describe the same work, so duplicates can be merged.
Real exampleOCLC WorldCat de-duplicationOCLC uses a machine learning model to detect duplicate bibliographic records in WorldCat and reports that it has extended the model to all formats, languages and scripts.
Reading handwritten and printed archives
Handwritten text recognition models turn scanned manuscripts and old print into searchable text. Archivists train or choose a model for a script and period, then correct a sample to measure accuracy.
Real exampleTranskribusTranskribus, run by the European cooperative READ-COOP, uses AI to read handwritten and printed historical documents and turn them into searchable text.
Searching digitised collections by image
Computer vision models extract photographs, maps and illustrations from digitised pages and represent them as embeddings. Readers can then browse visually similar images across a whole collection.
Real exampleNewspaper Navigator (Library of Congress)An experimental Library of Congress application, now retired, that used machine learning to support visual similarity search over more than 1.5 million images extracted from Chronicling America newspapers.
Can librarians learn machine learning?
Yes. Librarians and metadata specialists already understand controlled vocabularies, record quality and user needs, which is exactly what library ML projects depend on. Learn Python and pandas for records, then text classification, then embeddings and language models for search.
- Python for catalogue dataRead MARC or exported CSV records with Python, clean fields and count subjects, which is the first step of any library ML project.Learn it in Python
- Tables with pandasTurn records into a table of titles, descriptions and subject headings, and split it into training and test sets.Learn it in Data Tools
- Text classificationTrain multi-label classifiers on titles and abstracts and measure precision at k against cataloguers' headings.Learn it in Machine Learning
- Embeddings and language models for searchEmbeddings and retrieval-augmented generation power semantic search and question answering over collections.Learn it in Generative AI
Moving into machine learning for libraries
If you already work in the field
Librarians, cataloguers and metadata specialists already know the standards, vocabularies and quality checks a model must meet. Add Python and text classification. A realistic first move is to test a tool such as Annif on a sample of your own records and compare its suggestions with cataloguers' headings.
If you come from data science or ML
If you come from ML, learn MARC, authority files and controlled vocabularies such as LCSH, and how cataloguing decisions are made and reviewed. Work alongside a cataloguer, because evaluation depends on their judgement of which suggestions are acceptable.
Roles that do this workMetadata librarianDigital scholarship librarianData librarianLibrary systems developer
Tools you will use
| Tool | What you use it for |
|---|---|
| Annif | Train and serve automated subject-indexing models that suggest headings from a controlled vocabulary. |
| pymarc | Read, edit and write MARC 21 bibliographic and authority records in Python. |
| scikit-learn | Build multi-label subject classifiers and duplicate-record detectors from catalogue fields. |
| Sentence Transformers | Embed titles and abstracts for semantic search and related-item recommendations. |
| Apache Solr | Index catalogue records for full-text, faceted and vector search over a collection. |
Public data to practise on
- Open Library data dumps: Bulk works, editions and authors records with subjects; train subject suggestion or metadata-matching models.
- Library of Congress MDSConnect: Books All: Free bulk MARC bibliographic records from the Library of Congress; practise parsing and classifying catalogue data.
- Library of Congress MDSConnect: Subjects: Free LC Subject Headings authority records; use as the controlled vocabulary for automated indexing.
Your first project
Download the Open Library works dump, predict subject headings from titles and descriptions with TF-IDF and one-vs-rest logistic regression, and compare it with suggesting the most common subjects using precision at 5.
Common questions
Will machine learning replace cataloguers?
Current library systems use machine learning to suggest headings, flag duplicates and transcribe text, with people reviewing and correcting the results. Cataloguers' judgement is also what the models learn from and are tested against.
Do librarians need to code to use machine learning?
Some tools, such as Annif's web interface and Transkribus, can be used without coding. Building and evaluating your own models, or adapting them to your collection, needs Python.
How do libraries handle bias in automated metadata?
Models learn the patterns in past cataloguing, including outdated terms and gaps. Libraries review suggestions, test results across languages and subjects, and keep people responsible for the final record.
Courses for these skills
- Deep Learning with PyTorchStudy learned sequence representations.Syllabus available · lessons open as produced
- Generative AI: LLMs, RAG and AgentsConnect embeddings and retrieval to language-model applications.Syllabus available · lessons open as produced
- Data Tools: NumPy, pandas, Matplotlib and scikit-learnKeep catalogue identifiers and extraction metadata traceable.23 published lessons · all planned lessons open
- Practical Machine Learning in PythonBuild classification baselines and evaluate rare categories.51 published lessons · more lessons in production
- Maths for Machine LearningWork with norms, dot products and vector geometry.26 published lessons · all planned lessons open
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.