Semantic web ontology and metadata languages provide a new way to annotating and retrieving images. This poster, based on [4], considers the situation when a user is faced with an image repository whose content is complicated and semantically unknown. We show how ontologies can then be of help to the user in formulating the information need, the query, and the answers. We approach this problem through a case study based on an image collection on promotion ceremonies of the Helsinki University Museum. The actual implementation will be demonstrated at the poster session.
semantic web, image, information retrieval, ontology
A typical way to publish an image data repository is to create a keyword-based query [1] interface to an image database. Here the user may select filtering values or apply keywords to the different database fields, such as the "creator", "time", or to the content descriptions including classifications and free text documentation. More complex queries can be formulated, e.g., by using Boolean logic.
Keyword-based search methods suffer from several general limitations: A keyword in a document does not necessarily mean that the document is relevant, and relevant documents may not contain the explicit word. Synonyms lower recall rate, homonyms lower precision rate, and semantic relations such as hyponymy, meronymy, antonymy [3] are not exploited.
Keyword-based search is useful especially to a user who knows what keywords are used to index the images and therefore can easily formulate queries. This approach is problematic, however, when the user does not have a clear goal in mind, does not know what there is in the database, and what kind of semantic concepts are involved in the domain. Using the keyword-based approach would lead to the following problems:
Formulating the information need The user does not necessarily know what question to ask. One may only have general interest in the topic. How to help the user in focusing the interest within the database contents?
Formulating the query The user cannot necessarily figure out what keywords to use in formulating the search corresponding to her information need. How to help the user in formulating queries?
Formulating the answer Generating image hit lists for keywords would probably miss a most interesting aspect of the repository: the images are related to each other in many interesting ways. In our case, the ceremonial occasions follow certain patterns in place and time and the people and surroundings depicted in the images reoccur in different events. These semantical structures should somehow be exposed from the data to the audience. The goal of an ordinary museum visitor is often something quite different from trying to find certain images. The user wants to learn about the past and experience it with the help of the images.
We argue that semantic web technologies provide a promising new approach to these problems.
The problem of creating metadata for images has been of vital importance to art and historical museums when cataloging collection items and storing them in a digital form. Following approaches are commonly used in annotating images:
Keywords Controlled vocabularies are used to describe the images in order to ease the retrieval. In Finland, for example, the Finnish web thesaurus YSA [12] is used for the task augmented with museum- and domain specific keyword lists.
Classifications There are large classification systems, such as the ICONCLASS [11, 13] and the Art and Architecture Thesaurus [7], that classify different aspects of life into hierarchical categories. An image is annotated by a set of categories that describe it. For example, if an image of a seal depicting a castle could be related to classes "seals" and "castles". The classes form a hierarchy and are associated with corresponding keywords. The hierarchy enriches the annotations. For example, since castles are a subclass of "buildings", keyword "building" is relevant when searching images with a castle.
Free text descriptions Free text descriptions of the objects in the images are used. The information retrieval system indexes the text for keyword-based search.
Semantic web ontology techniques [2] and metadata languages [5] contribute to this tradition by providing means for defining class terminologies with well-defined semantics and a flexible data model for representing metadata descriptions. One possible step to take is to use RDF Schema for defining hierarchical ontology classes and RDF ontology. The ontology together with the image metadata forms an RDF graph, a knowledge base, which can facilitate new semantic information retrieval services. In our case application, we used this approach.
The ontologies form the core of our system and are used for three purposes:
Annotation terminology The ontological model provides the terminology and concepts by which metadata of the images is expressed.
View-based search The ontologies of the model, such as Events, Persons, and Places provide different views into the promotion content. Each view consists of classes and instances represented using the metaphor of a file system browser where the classes correspond to directories and instances to files. Queries are formulated by selecting resources from these views as the conjunction of the selections made. This view-based idea to information filtering along different indexing dimensions ("facets") is an adaptation of the HiBrowse system developed for a bibliographical information retrieval system [8].
Semantic browsing After finding a focus of interest, an image, the semantic ontology model together with image instance data can be used in finding out relations between the selected image and other images in the repository. These images are recommended to the user. They do not necessarily match the filtering query but that are likely to be of interest. They can for example, contain relatives of the Person in the photo.
By clicking on a recommended thumbnail photo, the large image in view is switched and a new set of recommended images is dynamically generated beneath it. This idea is vaguely related to topic-based navigation used in Topic Maps [5, 6] and the book recommendation facility in use at Amazon.com. Figure 1 illustrates the user interface of the system.
Our work showed that ontologies can be used not only for annotation and precise information retrieval [9, 10], but also for helping the user in formulating the information need and the corresponding query. This is important in applications such as the promotion exhibition, where the domain semantics are complicated and not necessarily known to the user. Furthermore, the ontology-enriched knowledge base of image metadata can be applied to constructing more meaningful answers to queries than just hit-lists. For example, in our implementation, the underlying knowledge base provided the user with a semantic browsing facility between related recommended images.
The major difficulty in the ontology based approach is the extra work needed in creating the ontology and the detailed annotations. We believe, however, that in many applications - such as in our case problem - this price is justified due to the better accuracy obtained in information retrieval and to the new semantic browsing facilities offered to the end-user. The trade-off between annotation work and quality of information retrieval can be balanced by using less detailed ontologies and annotations, if needed.
Thanks to Tero Halonen, Kati Heinämies, Robert Holmberg, Kim Josefsson, Pasi Lehtimäki, Tiina Metso, Eetu Mäkelä, Matti Nykänen, Taneli Rantala, and Jaana Tegelberg. Our work was partly funded by the National Technology Agency Tekes, Nokia, TietoEnator, the Espoo City Museum, and the Foundation of the Helsinki University Museum, and was supported by the National Board of Antiquities.