Re: Announcing OWLIM 5.0 - with new transaction mechanism, performance improvements, SPARQL 1.1 graph store protocol and more

That is an impressive list of new features Barry. Congratulations to the
OWLIM dev team with this new release!

Cheers,

Jeen
this message brought to you by Toronto airport free wifi :)
On Apr 20, 2012 8:47 AM, "Barry Bishop" <barry.bishop@ontotext.com> wrote:

>  Ontotext are pleased to announce the release of OWLIM version 5.0<http://www.ontotext.com/owlim>featuring a new transaction mechanism, performance improvements, SPARQL 1.1
> graph store protocol, integration with TopBraid Composer/Live<http://www.topquadrant.com/products/TB_Suite.html>and many other improvements. The single most important new feature is the
> new transaction management mechanism which allows for much *more reliable
> and efficient handling of workloads where queries from multiple clients are
> combined with frequent updates* of the data. As benchmark results<http://www.ontotext.com/owlim/benchmark-results/owlim-5>demonstrate, OWLIM 5.0 is
> *43% faster* than v.4.3 on the BSBM Explore and Update<http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/>scenario. As a result of several changes in the index structures, OWLIM now
> requires *between 25% and 70% less storage space*.
>
> Some of the most important improvements are listed below:
>
>    - *Transaction management and isolation mechanisms* have been
>    completely refactored. The previous strategy used lazy writing of modified
>    database pages, such that dirty pages were only flushed to disk when
>    further updates occur and no more memory is available. While extremely
>    fast, the problem with this approach is that there is a considerable
>    recovery time associated with replaying the transaction log after an
>    abnormal termination. The new mechanism uses two modes: 'bulk-loading'
>    (fast) with similar behaviour to previous versions and 'normal' (safe)
>    where database modifications are flushed to disk as part of the commit
>    operation. When running in safe mode, *database recovery is instant*and there is a
>    *significant improvement in concurrency between updates and queries*.
>
>
>    - *New context indices* can be used to improve query performance when
>    data is modelled using many named graphs. These are switched on and off
>    using a single configuration parameter enable-context-index
>
>
>    - The *SPARQL 1.1 Graph Store HTTP Protocol* is now supported
>    according to the W3C Working Draft<http://www.w3.org/TR/sparql11-http-rdf-update/>from the 12th May 2011. This provides a REST interface for managing
>    collections of graphs, using either directly or indirectly named graphs.
>
>
>    - *Sesame <http://www.openrdf.org>* *2.6.5* with many bug-fixes and
>    updates to bring SPARQL 1.1 Query<http://www.w3.org/TR/2012/WD-sparql11-query-20120105/>support up to the latest W3C Working Draft from the 5th January 2012.
>
>
>    - *Significant reduction in disk-space requirements* is achieved with
>    the following modifications:
>       - *Index compression* can now be used to reduce disk storage
>       requirements by using zip compression on database pages. This feature if
>       off by default, but can be switched on when creating a new repository. The
>       configuration parameter index-compression-ratio can be set to -1
>       (the default value indicating no compression) or a value in the range
>       10-50<https://confluence.ontotext.com/pages/createpage.action?spaceKey=OWLIMint&title=10-50&linkCreation=true&fromPageId=17596523>indicating the desired percentage reduction in page sizes. Any pages that
>       can not be compressed by the specified amount are stored uncompressed.
>       Therefore a compression ratio that is too aggressive will not bring many
>       benefits. Experiments have shown that for large datasets a value of about
>       30% is close to optimal and leads to a total disk space saving of around
>       50%.
>       - *Restructuring of the triple indices* has also led to a reduction
>       in disk-space requirements of around 18% independent of the compression
>       functionality
>       - *Entity compression* is a modification that reduces the storage
>       requirements for the lookup table that maps between internal identifiers
>       and resources. This is transparent to the user and happens automatically.
>       More disk space reductions are apparent using this version.
>
>
>    - A new *literal index* is created automatically for numeric and
>    date/time data-types. The index is used during query evaluation if a query
>    or a sub-query (e.g. union) has a filter that is comprised of a conjunction
>    of literal constraints, e.g. FILTER(?x >= 3 && ?y <= 5 && ?start >
>    "2001-01-01"^^xsd:date). Other patterns, including those that use negation,
>    will not use the index for this version of OWLIM.
>
>
>    - Tighter integration with TopQuadrant <http://www.topquadrant.com/>'s TopBraid
>    Composer <http://www.topquadrant.com/products/TB_Composer.html> (a
>    graphical development environment for modelling data) and TopBraid Live<http://www.topquadrant.com/products/TB_Live.html>(an enterprise SOA-capable Semantic Web application platform). Contact the OWLIM
>    team directly <owlim-info@ontotext.com> for details of how to obtain
>    the OWLIM plug-in.
>
>
>    - All *control queries now use SPARQL Update syntax* (used mostly to
>    control the Lucene-based full-text search, RDF Rank and geo-spatial
>    plug-ins). This has a number of advantages, namely:
>       - No special control query pseduo-graph is required by the
>       Replication Cluster master in order to identify control queries that must
>       be pushed to all worker nodes
>       - SPARQL Updates use the corresponding SPARQL update protocol, so
>       they can be automatically processed by load-balancers that examine URL
>       patterns
>       - It is more consistent with the SPARQL language, since these
>       'control queries' cause a change of state in OWLIM
>
>
>    - *Incremental Lucene-based full-text search index* for updating the
>    index for specific resources or all un-indexed resources. Using this
>    technique can avoid the more expensive approach of rebuilding the whole
>    index frequently.
>
>
>    - *Incremental RDF Rank* allows the RDF rank for specific resources to
>    be (re-)computed as directed by the user. This technique can avoid the more
>    expensive approach of rebuilding all RDF Rank values frequently.
>
>
>    - As well as the cache/index statistics, *performance analysis data*is now provided about currently executing queries including: how many
>    results have been returned so far, how long it has been executing, average
>    time to return each result, etc.
>
>
>    - The *getting started* application has been restructured so that it
>    now works with remote repositories.
>
> *Known problems with OWLIM 5.0*
>
>    - The behaviour of the 'include inferred' checkbox in the Sesame
>    Workbench is unpredictable when using OWLIM repositories.
>    - This version of OWLIM is *not backwardly compatible* with any
>    previous version. This means that images created with OWLIM 4.3 and before
>    will not work correctly with OWLIM 5.0 and must be re-created. There have
>    been a great many modifications to the storage files, indexing structures,
>    etc, and upgrade mechanisms have proven too complex and probably slower
>    than re-loading the database anyway. Please *do not attempt to upgrade
>    to OWLIM 5.0 unless you drop and recreate all databases*. A migration
>    tool, which allows for automated re-loading of data from any
>    Sesame-accessible repository, is provided to ease the transition.
>
> For further technical information and references to resolved technical
> issues, please refer to the Release notes<http://owlim.ontotext.com/display/OWLIMv50/OWLIM-SE+Release+notes>of the corresponding edition of OWLIM. Full documentation for all OWLIM
> editions is available online <http://owlim.ontotext.com> (click on the
> OWLIM 5.0 link on the left hand side).
>
> One can request further information and evaluation licences for OWLIM from
> here <http://www.ontotext.com/owlim#download>.
>
> The OWLIM team
> April 2012
>
>