RE: Requesting guidance on EXI fragment encryption and signature

Hi Paul,

Thank you for your interest in EXI and the effort to evaluate EXI with your
application.

As you may be aware, EXI incorporates a compression feature [1] that can
be optionally turned on for better compactness. EXI compression supports
streamability, which means that the a decoder only needs access to a block
of data before being able to start decoding the EXI stream into XML events
such as SAX. Similarly, an encoder can start sending the stream off on the
wire block by block, in stead of waiting for the complete stream to be
generated. Also, it is worth noting that EXI encoding with compression
feature on, works much faster than gzipped XML. Our evaluation [2] shows
that the average decoding speed of EXI (with compression on) was 9.2 times
faster than the average decoding speed of GZipped XML, and the average
encoding speed of EXI was 5.4 times faster than the average encoding speed
of Gzipped XML. This is of course in addition to the extra benefit of
compactness you can achieve over gzip.

The best configuration that makes most out of EXI generally depends on a
variety of factors such as the document size, the ability to use schemas
whether it be complete or partial, the affordability of compression. EXI
is designed so that it will likely be found the best choice over alternative
formats given any set of application requirements that constrain those
parameters.

The compression feature should be turned on if the application is tolerant
about the exercise of compression. Given you have included zlib in the
result set and EXI compression consumes less resources, it would be more
natural to include only those measurement results that were yielded
with compression turned on in the comparison chart alongside zlib.
Orthogonal to compression is the use of schemas. Schemas should be
used if there are schemas available in the use case. In addition, strict
interpretation of schemas should be used if the use case only expects
schema-valid instance documents, which nonetheless does not seem to
appear the case in your application.

Note also that the aligmnent option values "byte-alignment" and
"precompression" are meant to be used only in certain special circumstances.
The primary use of the alignment value "byte-alignment" is for debugging,
and the value "precompression" is used when compression capability is built
into the transport protocol, therefore, its use makes sense only when EXI is
used with external compression (such as gzip or zlib) to be applied
underneath. On the other hand, lexical preservation is useful only when
literal transmission of values are required, such as when the XML signature
is used with EXI. If those options values are used with no requirement that
motivates their respective use, it only leads to water down the compactness
benefits of EXI format, and that seems rather departed from the practical use
of EXI.

[1] http://www.w3.org/TR/2009/CR-exi-20091208/#compression
[2] http://www.w3.org/TR/exi-evaluation/

Hope it helps,

-taki


  _____

From: public-exi-request@w3.org [mailto:public-exi-request@w3.org] On Behalf Of Paul E. Jones
Sent: Tuesday, October 20, 2009 7:02 PM
To: 'Snyder, Sheldon (LT)'; public-exi@w3.org; 'Brutzman, Donald (Don) (CIV)'; 'McGregor, Donald (Don) (CIV)'
Subject: RE: Requesting guidance on EXI fragment encryption and signature



Don,



I do not have guidance, but the questions are nonetheless of interest to me.  So, if you do not mind, let me make a few statements
and then ask a few questions for clarification.



I also looked at compressing EXI for an application I am working on, but decided that compression (e.g., such as with zlib) was not
desirable, as I would lose the ability to manipulate the stream on the wire: I really appreciate that attribute of both XML and EXI.



We looked at a few modes of EXI as compared to using zlib (without EXI at all) and came up with this:

http://www.packetizer.com/xml/xml_compression.html



It would be interesting to see other comparisons.  The net is that "strict compressed" yielded the best results, but we do not
necessarily have the schema in hand in our case, so that was out.  Zlib did extremely well for XML documents between 300 bytes and
8K.  For smaller messages, EXI alone (without compression) did better.  What was interesting was that above 8K, it seemed that all
modes (including zlib compression) largely converged in size.



I am curious: will compression be necessary in your case, or would EXI alone be sufficient?



Is the purpose for using EXI on only a part of the document to allow part of the document to be visible by entities that need to
know a bit about the contents without decoding it?  If you have large documents and use any form of EXI, wouldn't using EXI across
the whole XML document be workable?  Inspecting the EXI binary seems to be relatively simple: at any rate, one is going to employ a
parser one way or the other.



Is the purpose to allow one to apply a signature over the EXI parts only, with the unsigned parts outside?  In other words, you're
defining a "container", of sorts, to hold the real information, while placing the signature that applies to that data outside in
normal XML.  If so, s there any reason to not simply define such en explicit wrapper?  The signature issue is really a big deal and
I've not been to happy with the existing options.  So, the line of questioning is interesting - and I suspect to a number of people.



Paul



From: public-exi-request@w3.org [mailto:public-exi-request@w3.org] On Behalf Of Snyder, Sheldon (LT)
Sent: Tuesday, October 20, 2009 8:30 PM
To: public-exi@w3.org; Brutzman, Donald (Don) (CIV); McGregor, Donald (Don) (CIV)
Subject: Requesting guidance on EXI fragment encryption and signature



EXI Working Group:



Requesting guidance on EXI fragment encryption and signature:

1.      Sometimes want to compress only part of an XML document and retain the results in context within the new document.

2.      XML signature and XML encryption require XML inputs.

3.      Therefore an EXI document or EXI fragment cannot be signed or encrypted because they are not XML.

4.      Need formal definition on how to embed an EXI fragment in an XML document so that it can be signed and or encrypted.



Example:

Want to EXI the SequenceOfEvents and retain in the XML document for digital signature and encryption:



<?xml version="1.0" encoding="UTF-8"?>

<PlanOfDay>

    <dayOfWeek date="Monday 18 October 2009"/>

    <Title>Some Tile for the day's Events</Title>

    <SequenceOfEvents>

        <Event>The first event of the day</Event>

        <Event>The second event of the day</Event>

        <Event>The third event of the day</Event>

    </SequenceOfEvents>

</PlanOfDay>



....



<?xml version="1.0" encoding="UTF-8"?>

<PlanOfDay>

    <dayOfWeek date="Monday 18 October 2009"/>

    <Title>Some Tile for the day's Events</Title>

            EXI of SequenceOfEvents

</PlanOfDay>



Could do but:

<?xml version="1.0" encoding="UTF-8"?>

<PlanOfDay>

    <dayOfWeek date="Monday 18 October 2009"/>

    <Title>Some Tile for the day's Events</Title>

    <SequenceOfEvents>

            EXI Fragment of SequenceOfEvents

    </SequenceOfEvents>

</PlanOfDay>



However, once the XML is compressed to EXI it is no longer effectively compressible because EXI will have few if any redundant
bytes; no compression algorithm will deliver good results.



Some potential solutions of consideration:

1.      is a base64 formatting such as Message Transmission Optimization Mechanism (MTOM)

2.      XML-binary Optimized Packaging (XOP)

3.      Williams, Jeffrey S., Document-Centric XML Encryption and Authentication for Coalition Messaging, Masters Thesis, Naval
Postgraduate School, Monterey California, September 2009.



For additional guidance on the intent of this question please email Don Brutzman: Brutzman@nps.edu