Use Case: "Art-image collection"

Contributor: Guus Schreiber

Context

We are working on semantic annotations of images of art objects. The purpose is to support both indexing and search through ontologies. There are many knowledge sources for art. We focus here on two of these:

The VRA 3.0 standard for image descriptions, which is basically a refinement of Dublin Core for art-image annotation
The Art and Architecture Thesaurus (AAT) constructed by the Getty Foundation, which provides a highly structured hierarchy of some 120.000 terms to describe art objects (art categories, materials, styles, color, ....).

We want to use OWL to represent the image description template provided by VRA and to link every data element of VRA to the subtrees of the AAT hierarchy where the "fillers" of the data element can be found. For example, we want to link the VRA data element "style/period" to the AAT subtrees representing styles and periods.

In addition, we want to express in the ontology additional knowledge. For example, if an indexer selects the value "Late Georgian" for the style/period of (say) an antique chest of drawers, we want to be able to infer that the data element "date.created" should have a value between 1760 and 1811 A.D. and that the "culture" is British. Availability of this type of background knowledge increases significantly the support that can be given for indexing as well as for search.

Task

Indexing and searching a digital image collection

Example domain

Museum collection of images of antique furniture

Typical users

Museum personnel involved in indexing images.
Lay person with some basic knowledge of the domain, looking for some piece of antique

Ontology samples

Representing the AAT color hierarchy

In our ontology we want to express what AAT terms can act as values for the VRA data element "color". AAT has an elaborate hierarchy for colors, which is structured more or less like this:

<color>
  <chromatic color>
    pink
      vivid pink 
      strong pink
      ....
      <intermediate pink>
        purplish pink
	  brilliant purplish pink
	  ....
	yellowish pink
	  ....
	brownish pink
    (etc.)
  <neutral color>
    white
    gray
      light gray
      ....
    black

The terms of type "<label>" are what AAT calls 'guide terms'. Their purpose is to provide structure to the hierarchy. When we specify a value restriction for the slot "color" of an image description template we ideally just want to say that any subclass of the <color> hierarchy can be used as slot filler.

REQUIREMENT: Class as slot filler. The corresponding range constraint should typically specify of what superclass(es) the slot fillers should be a subclass.

BTW: An assumption of our work is that in order to be successful we need to build on the (semi-)ontologies already available (like AAT), and that we will have to map these onto a representation in OWL. It is unrealistic to assume we can redo large-scale efforts like AAT.

Representing an aggregate structure

When we want to index an object such as an antique chest of drawers, there is almost always a need to represent the part-of structure of the object. For example, the we want to assign a style value to the feet of a chest, e.g. "bun feet". In our view the working group should seriously consider introducing some (limited form) of aggregation into OWL. If you just represent this as another slot/relation, you lose much of the semantics. This is likely to be a requirement from UML people as well (aggregation has a prominent place in UML class models).

REQUIREMNT: Possibility to specify that a property denotes a part-whole (containment/aggregation) relation.

Definitional knowledge

Let's for the moment assume we can represent AAT and VRA in OWL. For effective search support we need to add domain knowledge to this ontology. This knowledge typically takes the form of inter-slot constraints within the image description template. One example:

  style/period = "Late Georgian"
  =>
  culture = "British" AND
  date.created = between 1760 and 1811

[Style/period, culture and date.created are all VRA data elements defined as slots for our art-object description template.]

We could not define this constraint in RDFS. Sean Bechofer (University of Manchester) provided a DAML+OIL solution (details of data-type representation and URIs left out):

<daml:Restriction>
 <daml:onProperty rdf:resource="some-URL#style"/>
  <daml:hasClass>
   <daml:Class rdf:about="some-URL#Late Georgian"/>
  </daml:hasClass>
 <rdfs:subClassOf>
  <daml:Class>
   <daml:intersectionOf rdf:parseType="daml:collection"/>
    <daml:Restriction>
     <daml:onProperty rdf:resource="some-URL#date"/>
     <daml:hasClass>
      <daml:Class rdf:about="some-URL#1760-1811"/>
      </daml:hasClass>
     </daml:Restriction>
     <daml:Restriction>
      <daml:onProperty rdf:resource="some-URL#culture"/>
      <daml:hasClass>
       <daml:Class rdf:about="some-URL#British"/>
      </daml:hasClass>
     </daml:Restriction>
    </daml:intersectionOf>
  </daml:Class>
 </rdf:subClassOf>
</daml:Restriction>

So, all Late-Georgian things are subclasses of British things and all things created between 1760-1811. This is similar to what is called multiple specialization in data modelling.

Two issues arise here:

The syntax is really awful.
For most users this will not be an intuitive way of defining definitional knowledge. Can OWL provide a more natural way?

REQUIREMENT: Natural way to define definitional constraints.

Default knowledge

This is in fact the most common form of domain knowledge in the example domain. One sample of default knowledge:

IF type  "chest of drawers" AND
   style/period = Late-Georgian
THEN (this typically suggests)
   material.main = mahogany

This knowledge is crucial for real semantic queries, e.g. a user query for "antique mahogany storage furniture" could match with images of Late Georgian chests of drawers, even if nothing is said about wood type in the image annotation.

One similar example from a different image search domain: suppose a person wants to search for images of a "red ape". Most photos of orang-utans will match this query. However, one cannot expect the indexer of every orang-utan photo to state explicitly the color of the ape (this also leads to unwanted inter-indexer variability). You really want them to link the photo to the class "orang-utan", and possibly only specify the color if it is not red/orange (old animals can get brown/grey, there are albino orang-utans, etc.). If we do not allow the specification of default color values for orang-utans in general, we will not find these matching photos.

REQUIREMENT: Possibility to represent default knowledge

Summary of OWL requirements

Class as slot filler. The corresponding range constraint should typically specify of what superclass(es) the slot fillers should be a subclass.
Possibility to specify that a property denotes a part-whole (containment/aggregation) relation.
Natural way to define definitional constraints.
Possibility to represent default knowledge.

References

The Art and Architecture Thesaurus http://shiva.pub.getty.edu.
Visual Resources Association Standards Committee. VRA core categories, version 3.0. Technical report, Visual Resources Association, July 2000. http://www.gsd.harvard.edu/~staffaw3/vra/vracore3.htm.