Using AI/Machine Learning to Extract Data from Japanese American Confinement Records

As part of a Japanese American Confinement Sites (JACS) grant-funded project, supported by the National Park Service, the Bancroft Library at UC Berkeley is digitizing nearly 210,000 pages of War Relocation Authority (WRA) Form 26 individual records of Japanese Americans incarcerated during WWII. The library has partnered with Doxie.AI to develop a customized machine learning pipeline for extracting structured data from these records A number of challenges have arisen due to variability in content, structure, and placement of text on the page, and the presence of a wide variety of characteristics in the archival records that produce visual noise. This has prompted an iterative and dynamic approach to process records by camp. By blending library staff’s content and domain knowledge with the technical expertise of Doxie.AI, the project team is meeting or exceeding baseline targets for accuracy rates across the majority of fields. Additionally, ethical issues pertaining to the digital release of this data and the digitized records in their entirety will be explored with a Community Advisory Group, as the library seeks to establish a responsible digital curation plan for these resources. In alignment with Collections as Data principles that encourage the responsible computational use of special collections, this project represents a crucial opportunity to explore new methods for enhancing access to our growing digital special collections.