Many organizations maintain extremely large-scale image collections. The National Aeronautics and Space Administration (NASA) is such an example, which has hundreds of thousands of images, stored in different formats, levels of availability and resolution, and with associated descriptive information at various levels of detail and formality. Such an organization also generates thousands of images on an ongoing basis that are collected and cataloged. Thus, a mechanism is needed to catalog all the different types of image content across various domains. Information about both the image itself (e.g., its creation date, dpi, source) and about the specific content of the image is required. Additionally, the associated metadata must be maintainable and extensible so that associated relationships between images and data can evolve cumulatively. Lastly, management functionality should provide mechanisms flexible enough to enforce restriction based on content type, ownership, authorization, etc.
One possible solution for such image management requirements is an annotation environment that enables users to annotate information about images and/or their regions using concepts in ontologies (OWL and/or RDFS). More specifically, subject matter experts will be able to assert metadata elements about images and their specific content. Multimedia related ontologies can be used to localize and represent regions within particular images. These regions can then be related to the image via a depiction/annotation property. This functionality can be provided, for example, by the MINDSWAP digital-media ontology (to represent images, image regions, etc.), in conjunction with FOAF (to assert image depictions). Additionally, in order to represent the low level image features of regions, the aceMedia Visual Descriptor Ontology can be used.
In order to describe the content of such images, a mechanism to represent the domain specific content depicted within them is needed. For this use case, domain ontologies that define space specific concepts and relations can be used. Such ontologies are freely available and include, but are not limited to the following:
As discussed above, this scenario requires the ability to state that images (and possibly their regions) depict certain things. For example, consider a picture of the Apollo 7 Saturn shuttle launch. One would want to make assertions that include that the image depicts the Apollo 7 launch, the Apollo 7 Saturn IB space vehicle is depicted in a rectangular region around the rocket, the image creator is NASA, etc. One possible way to accomplish this is to use a combination of various multimedia related ontologies, including FOAF and the MINDSWAP digital-media ontology. More specifically, image depictions can be asserted via a depiction property (a sub-property of foaf:depiction) defined in the MINDSWAP Digital Media ontology. Thus, images can be semantically linked to instances defined on the Web. Image regions can defined via an ImagePart concept (also defined in the MINDSWAP Digital Media ontology). Additionally, regions can be given a bounding box by using a property named svgOutline, allowing localizing of image parts. Essentially SVG outlines (SVG XML literals) of the regions can be specified using this property. Using the Dublin Core standard and the EXIF Schema more general annotations about the image can be stated as well, including its creator, size, etc. A subset of these sample annotations are shown in an RDF graph below in Figure 1.
Figure 1 illustrates how the approach links metadata to the image:
http://www.mindswap.org/2005/owl/digital-medial#depictshttp://www.mindswap.org/2005/owl/digital-medial#hasRegionhttp://www.mindswap.org/2005/owl/digital-medial#svgOutline and an SVG snippet
Additionally, the entire annotations of the Apollo 7 launch are shown below in RDF/XML.
<rdf:RDF
xmlns:j.0="http://www.w3.org/2003/12/exif/ns/"
xmlns:j.1="http://www.mindswap.org/2005/owl/digital-medial#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:j.2="http://semspace.mindswap.org/2004/ontologies/System-ont.owl#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns:owl="http://www.w3.org/2002/07/owl#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:j.3="http://semspace.mindswap.org/2004/ontologies/ShuttleMission-ont.owl#"
xml:base="http://example.org/NASA-Use-Case" >
<rdf:Description rdf:about="A0">
<j.1:depicts rdf:resource="#Saturn_1B"/>
<rdf:type rdf:resource="http://www.mindswap.org/~glapizco/technical.owl#ImagePart"/>
<rdfs:label>region2407</rdfs:label>
<j.1:regionOf rdf:resource="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg"/>
<j.1:svgOutline>
<svg xml:space="preserve" width="451" heigth="640" viewBox="0 0 451 640">
<image xlink:href="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg" x="0" y="0" width="451" height="640" />
<rect x="242.0" y="79.0" width="46.0" height="236.0" style="fill:none; stroke:yellow; stroke-width:1pt;"/>
</svg>
</j.1:svgOutline>
</rdf:Description>
<rdf:Description rdf:about="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg">
<j.0:imageLength>640</j.0:imageLength>
<dc:date>10/11/1968</dc:date>
<dc:description>Taken at Kennedy Space Center in Florida</dc:description>
<j.1:depicts rdf:resource="#Apollo_7_Launch"/>
<j.1:hasRegion rdf:nodeID="A0"/>
<dc:creator>NASA</dc:creator>
<rdf:type rdf:resource="http://www.mindswap.org/~glapizco/technical.owl#Image"/>
<j.0:imageWidth>451</j.0:imageWidth>
</rdf:Description>
<rdf:Description rdf:about="#Apollo_7_Launch">
<j.3:launchDate>10/11/1968</j.3:launchDate>
<j.3:codeName>Apollo 7 Launch</j.3:codeName>
<j.3:has_shuttle rdf:resource="#Saturn_1B"/>
<rdfs:label>Apollo 7 Launch</rdfs:label>
<j.1:depiction rdf:resource="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg"/>
<rdf:type rdf:resource="http://semspace.mindswap.org/2004/ontologies/ShuttleMission-ont.owl#Launch"/>
</rdf:Description>
<rdf:Description rdf:about="#Saturn_1B">
<rdfs:label>Saturn_1B</rdfs:label>
<j.1:depiction rdf:nodeID="A1"/>
<rdfs:label>Saturn 1B</rdfs:label>
<rdf:type rdf:resource="http://semspace.mindswap.org/2004/ontologies/System-ont.owl#ShuttleName"/>
<j.1:depiction rdf:nodeID="A0"/>
</rdf:Description>
</rdf:RDF>
In order to represent the low level features of images, the aceMedia Visual Descriptor Ontology can be used. This ontology contains representations of MPEG-7 visual descriptors and models Concepts and Properties that describe visual characteristics of objects. For example, the dominant color descriptor can be used to describe the number and value of dominant colors that are present in a region of interest and the percentage of pixels that each associated color value has.
Existing toolkits, such as [PhotoStuff] and [M-OntoMat-Annotizer], currently provide graphical environments to accomplish the annotation tasks mentioned above. Using such tools, users can load images, create regions around parts of the image, automatically extract low-level features of selected regions (via M-OntoMat-Annotizer), assert statements about the selected regions, etc. Additionally, the resulting annotations can be exported as RDF/XML (as shown above), thus allowing them be shared, indexed, and used by advanced annotation-based browsing (and searchable) environments.