|
|
--- |
|
|
title: DOAB Metadata Extraction Evaluation |
|
|
emoji: 📚 |
|
|
colorFrom: blue |
|
|
colorTo: purple |
|
|
sdk: docker |
|
|
pinned: false |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
# VLM vs Text: Extracting Metadata from Book Covers |
|
|
|
|
|
**Can Vision-Language Models extract metadata from book covers better than text extraction?** |
|
|
|
|
|
## TL;DR |
|
|
|
|
|
**Yes, significantly.** VLMs achieve ~97% accuracy vs ~75% for text extraction on title extraction, and ~80% vs ~71% on full metadata extraction. |
|
|
|
|
|
## The Task |
|
|
|
|
|
Extracting metadata from digitized book covers is a common challenge in libraries, archives, and digital humanities projects (GLAM sector). We compared two approaches: |
|
|
|
|
|
1. **VLM (Vision)**: Send the cover image directly to a Vision-Language Model |
|
|
2. **Text Extraction**: Extract text from the image first, then send to an LLM |
|
|
|
|
|
## Results |
|
|
|
|
|
### Title Extraction (simpler task) |
|
|
|
|
|
| Approach | Average Accuracy | |
|
|
|----------|-----------------| |
|
|
| **VLM** | **97%** | |
|
|
| Text | 75% | |
|
|
|
|
|
### Full Metadata (title, subtitle, publisher, year, ISBN) |
|
|
|
|
|
| Approach | Average Accuracy | |
|
|
|----------|-----------------| |
|
|
| **VLM** | **80%** | |
|
|
| Text | 71% | |
|
|
|
|
|
VLMs consistently outperform text extraction across both tasks. |
|
|
|
|
|
### Why VLMs Win |
|
|
|
|
|
Book covers are **visually structured**: |
|
|
- Titles appear in specific locations (usually top/center) |
|
|
- Typography indicates importance (larger = more likely title) |
|
|
- Layout provides context that pure text loses |
|
|
|
|
|
Text extraction flattens this structure, losing valuable spatial information. |
|
|
|
|
|
## Models Evaluated |
|
|
|
|
|
**VLM Models**: |
|
|
- Qwen3-VL-8B-Instruct (8B params) |
|
|
- Qwen3-VL-30B-A3B-Thinking (30B params) |
|
|
- GLM-4.6V-Flash (9B params) |
|
|
|
|
|
**Text Models**: |
|
|
- gpt-oss-20b (20B params) |
|
|
- Qwen3-4B-Instruct-2507 (4B params) |
|
|
- Olmo-3-7B-Instruct (7B params) |
|
|
|
|
|
**Interesting finding**: Qwen3-VL-8B achieves 94% even when used as a text-only model, suggesting it's generally better at this task regardless of modality. |
|
|
|
|
|
## Interactive Features |
|
|
|
|
|
- **Task selector**: Switch between Title Extraction and Full Metadata results |
|
|
- **Model size vs accuracy plot**: Interactive scatter plot showing efficiency |
|
|
- **Leaderboard**: Filter by VLM or Text approach |
|
|
|
|
|
## Technical Details |
|
|
|
|
|
- **Dataset**: [DOAB Metadata Extraction](https://huggingface.co/datasets/biglam/doab-metadata-extraction) (50 samples) |
|
|
- **Evaluation Framework**: [Inspect AI](https://inspect.aisi.org.uk/) |
|
|
- **Scoring**: |
|
|
- Title: Flexible matching (handles case, subtitles, punctuation) |
|
|
- Full Metadata: LLM-as-judge with partial credit |
|
|
- **Logs**: [davanstrien/doab-title-extraction-evals](https://huggingface.co/datasets/davanstrien/doab-title-extraction-evals) |
|
|
|
|
|
## Replicate This |
|
|
|
|
|
The evaluation logs are stored on HuggingFace and can be loaded directly: |
|
|
|
|
|
```python |
|
|
from inspect_ai.analysis import evals_df |
|
|
|
|
|
df = evals_df("hf://datasets/davanstrien/doab-title-extraction-evals") |
|
|
``` |
|
|
|
|
|
## Why This Matters for GLAM |
|
|
|
|
|
Libraries and archives have millions of digitized documents where metadata is incomplete or missing. VLMs offer a promising approach for: |
|
|
|
|
|
- **Catalog enhancement**: Fill gaps in existing records |
|
|
- **Discovery**: Make collections more searchable |
|
|
- **Quality assessment**: Validate existing metadata |
|
|
|
|
|
This evaluation demonstrates that domain-specific benchmarks can help identify the best approaches for cultural heritage tasks. |
|
|
|
|
|
--- |
|
|
|
|
|
*Built with [Marimo](https://marimo.io) | Evaluation framework: [Inspect AI](https://inspect.aisi.org.uk/)* |
|
|
|
