KI für Citizen Science in der Schweiz

How can artificial intelligence support Citizen Science? At the beginning of the year, we at catta thought about it. Here’s a short update, cliffhanger: there's news 😉  

Practical use of AI in classification projects of archives 

After the research, we wanted to start a first test right away. We asked ourselves, what could such AI support look like in practice?  

In our Citizen Science introductory workshops for museums and libraries, there is often the desire to classify, analyze, assign digitized images or objects to a location, find certain items on them, or similar tasks  

Our consideration from that: It’s exciting, but who wants to (voluntarily!) look through a few thousand images? But what if the images are automatically pre-processed, for example, sorted by themes or specific objects? This makes the work for Citizen Scientists much more exciting, as the drudge work falls away and they can focus more on the content. 

Test phase: Image description and tagging with openAI 

We asked two archives if they could provide us with some of their images so we could test how well the image description and tagging of digitized photos works with publicly accessible AI tools (we used openAI). Many thanks to Myrta Gegenschatz from the State Archive of Appenzell Ausserrhoden as well as Dorothée Guggenheimer and Oliver Ittensohn from the City Archive and Vadian Collection of the Local Community of St. Gallen and the Political Community of St. Gallen for engaging in this AI experiment with us. 

With the help of a script, AI openAI was supposed to generate keywords and an image description for each image. Most of the images were black and white photos – a particular challenge.  

AI image analysis: The first insights 

We were pleasantly surprised by how detailed the image descriptions were. The AI sometimes found people in the black and white images that we only discovered after looking several times. The tagging with general keywords also worked superbly in these first tests. Sometimes the AI even draws its own conclusions. In the following example, it concluded a celebration (or national pride) from “several Swiss flags on buildings.” 

Misinterpretations by AI 

The information from this first test, however, is still to be taken with caution. Sometimes there are complete misinterpretations of images.

Whenever expertise is required, it becomes difficult. In some examples, the AI admits it is not quite sure. In other cases, it simply claims, for instance, that a skeleton in a showcase must be a dinosaur.

Similarly, unfortunately (still) not possible or faulty in our experiment are assessments of exact locations and years or epochs.

Joke or unintentionally funny

And sometimes the AI is simply funny – whether intentional or not is beside the point.

Conclusion: Collaboration between AI and Citizen Scientists

A software like openAI does not yet replace humans when it comes to creating a final image description. However, it offers useful foundations for Citizen Science, allowing humans to add details and expertise:

• Thematic pre-sorting of large volumes of digitized images
• Search for specific objects in an image collection, e.g., “dog”
• General image description

What’s next?

The AI journey isn’t over yet. We’re putting our heads together again to think about which citizen science project we could use the tool for, or which project we could launch with it. Perhaps we could have the AI analyse the food waste photos for future “Aufgabeln!” project implementations? Perhaps we could search art archives for images containing historical evidence of certain animal species to aid biodiversity research?

It’s all still very exciting, so stay tuned and why not sign up for our newsletter right away (you’ll only receive it when we’ve got something to say 😉).

Got any questions or ideas of your own? Drop us a line at hoi@catta.ch

Images: City Archives and Vadian Collection of the Local Citizens’ Community of St. Gallen and the Political Municipality of St. Gallen

Text by: Pia Viviani