BiblioScrape and BiblioMatch
The project
The EMERGENCE research team maintains OE-BaRD, a database of publications, people, and letters related to 19th-century Old English studies. A lot of bibliographical data collection and matching was still done manually, which made expanding the database slow and error-prone. Our product, BiblioScrape and BiblioMatch, helps researchers search RI-OPAC, export bibliographical metadata to CSV, and compare it with existing OE-BaRD data. It uses normalization and fuzzy matching to find likely duplicate or related records, then lets the user review uncertain matches before exporting the results. This saves the client time, reduces repetitive manual work, and helps keep the database more consistent. It also gives researchers a clearer workflow for collecting, checking, and merging bibliographical records.
Some records matched perfectly. Others clearly wanted to keep their 19th-century mystery.
The customer
Our client was the EMERGENCE project at the Leiden University Faculty of Humanities, funded by the European Research Council. The project researches the history of Old English studies in Continental Europe. Communication with the client was useful and practical. We had regular meetings, showed progress during the project, and adjusted the tool based on feedback about metadata fields, matching behavior, and usability.
The client brought the bibliography; the Tech Wizards brought the spellbook for matching it.
The team
Our team was Tech Wizards, a group of six students. We worked with Scrum roles, including a Scrum Master and Product Owner, and divided the work across scraping, parsing, matching, the web interface, testing, documentation, and project management. We collaborated through sprints, code reviews, meetings, and shared backlog planning. One major challenge was RI-OPAC’s anti-scraping protection, which forced us to adapt the architecture and add a local database workflow. We are most proud of delivering an end-to-end tool that can search, export, match, review, and merge bibliographical records.
Some teammates scraped, some matched, some tested, and somehow the spell compiled.
The technologies
Languages:
- Python: backend, scraping, matching, data processing, and web interface
Important frameworks and libraries:
- Flask: local web interface
- SQLite: local RI-OPAC database workflow
- pandas: CSV processing
- RapidFuzz: fuzzy matching of titles and authors
- Playwright, requests, aiohttp: retrieving data from RI-OPAC
Tooling:
- uv: Python dependency management
- Ruff: linting and formatting
- ty: type checking
- pytest: automated testing
Deployment:
- The application runs locally through a Flask server. There was no public deployment, since the tool is intended for the client’s research workflow.