AI for Accessible Alternative Descriptions of Artworks and Specialist Graphics

Learning materials often rely on images such as illustrations, archival photographs, diagrams, and formulas. When those visuals are not described, some people with disabilities lose access to important information. Sages is carrying out an EU co-financed project to automatically generate alternative descriptions for specialist graphics at a quality level that can genuinely support the digital accessibility of educational materials.

In the recent period, research and development work focused on two areas:

• methods for automatically generating alternative descriptions for museum objects, including artworks such as prints, photographs, or sculptures, as well as historical artifacts such as furniture or coins,
• methods for automatically generating alternative descriptions for structural chemical formulas, that is, graphical representations of their structure.

The work included collecting training and test data using open educational and cultural resources as well as materials obtained in cooperation with the Archeology of Photography Foundation, preparing annotation instructions, adapting the annotation environment, and analyzing and comparing various methodological approaches that will become the basis for our proprietary solutions.

First, we prepared the resource base needed for further work. We identified and obtained source materials for two types of visual content and then organized them for model training and testing. We also reviewed alternative descriptions in the original collections, paying attention to their strengths and weaknesses in relation to good practices such as clarity, conciseness, and relevance. At the same time, we adapted the working environment so that the process of creating and further processing descriptions would be structured and scalable.

Second, we refined our qualitative approach. As part of the project, we defined the criteria for a good alternative description and the principles for creating reference descriptions. For museum objects, our starting point was existing guidance on audio description and extended descriptions of artworks published by cultural and educational institutions in Poland and abroad, as well as less formal guides prepared by blind people and by people describing artworks for their own needs. We standardized this knowledge and adapted it for use in the automation process. By contrast, work on chemical formulas required us to develop our own guidelines because there were no sufficiently specific materials for this type of content.

Third, we moved into the substantive work on the data itself. For artworks, we launched annotation at two levels: short alternative descriptions, needed for a screen-reader user to identify the basic characteristics of an object, and longer descriptions intended to convey the experience of engaging with the object as fully as possible. For structural formulas, we prepared extended descriptions that primarily help users understand the spatial arrangement of atoms and bonds while also conveying other important graphically expressed information. The first series of descriptions helped us establish a repeatable workflow. In the area of chemical formulas, we also reviewed methods and tools and identified the limitations of general-purpose language models that justify the development of specialist solutions.

AI as a bridge to cultural resources

As part of the work on art descriptions, an important step was obtaining and organizing source materials for building the dataset. At this stage, we used, among other things, unique collections of photographs, gouaches, and studies from the Archeology of Photography Foundation, which was the first organization in Poland to begin caring for the legacies of Polish photographers. FAF archives supported the preparation of examples for annotation and later testing of the approach. This makes it possible to design solutions that take into account the specific nature of diverse visual content and help build alternative descriptions with high usability.

Accessibility of visual resources is not just about technology trends. Above all, it is about genuinely expanding access to culture and education, says Mikolaj Chmielinski, archive specialist at the Archeology of Photography Foundation. We can already see that cooperation within this initiative, which develops the automatic creation of alternative descriptions, can lead to faster and more efficient sharing of artworks with people with visual impairments and older users. At the moment, cultural institutions do this manually and only for a small fraction of their collections. This project is an important direction that combines social responsibility with innovation, with technology working for the benefit of underserved groups.

What comes next: further research and work

In the next stages, the work will focus on expanding the datasets with initial AI support and on testing and refining the methods used so that the generated descriptions align more closely with the adopted guidelines and overall quality standards. At the same time, we are planning the next workstream on diagrams, which will begin with a review of current solutions and technology trends in this area.

We hope that delivering solutions that automate the preparation of descriptions will increase the number of resources available to people with visual impairments, especially where insufficiently described specialist visual materials remain a barrier to equal access to knowledge and culture.

Sages supports cultural institutions, research centers, and universities in implementation work in the areas of digital accessibility and AI-driven automation. We invite you to contact us regarding cooperation, demonstrations of the approach, and implementation activities.

_

The project entitled “Development of specialized algorithms that automatically create accessible alternative versions of specialist graphics for people with visual impairments in order to automatically ensure the accessibility of digital educational materials” is co-financed by the European Regional Development Fund under Priority I “European Funds for a more competitive and smarter Mazovia”, Measure 1.1 “Research, development and innovation of enterprises”, Project type “Research and development infrastructure of enterprises” of the European Funds for Mazovia 2021-2027 program.

Project value: PLN 4,887,450.00. European Funds contribution: PLN 2,432,225.00.