Weiwei Liu
PhD student in Data Science and Artificial Intelligence
The project: “Application of Computer Vision in the Digitization of Herbarium Specimen”
PhD Supervisor: Stefano Martellos, Associate Professor, University of Trieste
PhD Advisor: Jorge Calvo-Zaragoza, Full Professor, University of Alicante
Highlights:
· 1. Fine-tuning vision-language Transformer (VLT) models for automated metadata extraction from digitized herbarium specimens
· 2. Investigating the impact of background masking and data diversity on the generalization of fine-tuned models across herbarium collections
· 3. Exploring whether generalized image representations obtained by replacing the visual encoder in VLT model with a general-purpose DINO encoder can improve the transcription accuracy
Project description:
Herbarium collections represent invaluable resources for biodiversity research. With the advancement of digitization initiatives, millions of herbarium specimens have become accessible as high-resolution images. However, much of their associated metadata is recorded on handwritten and printed labels and still needs to be extracted. Manual transcription requires substantial manpower and time devotion, creating a significant bottleneck in the digitization.
This PhD project investigates the application of computer vision and deep learning techniques to automatically extract metadata from digitized herbarium specimen labels. The primary objective is to develop accurate and robust transcription models that can generalize across heterogeneous herbarium collections.
The initial research focused on fine-tuning Donut, an end-to-end vision-language Transformer (VLT), using structured metadata extracted from specimen images as training targets. Previous experiments demonstrated its potential for automated transcription but also revealed considerable performance differences across collections. To investigate these limitations, experiments involving visual encoder replacement and background masking have been conducted. The findings suggest that successful metadata extraction requires visual representations that balance the preservation of fine-grained textual information with the suppression of irrelevant background information.
Ultimately, this research seeks to advance the automation of herbarium digitization. The expected outcomes will contribute to more scalable and reliable metadata extraction, supporting the accessibility and scientific reuse of biodiversity collections.