W3C home > Mailing lists > Public > public-exi@w3.org > October 2008

Re: [EXI] Partial flush

From: Yuri Delendik <yury_exi@yahoo.com>
Date: Wed, 1 Oct 2008 18:31:46 -0700 (PDT)
To: Daniel Peintner <daniel.peintner@gmail.com>, public-exi@w3.org
Message-ID: <172201.12257.qm@web83202.mail.mud.yahoo.com>

Hi Daniel,
Thank you for suggestion. As EXI user I would expect maximum compression and byte-alignment does not provide that.
I ran EXI compression on weblog.xml (2,648,181 bytes) from EXI testing framework data using bit and byte alignment, and compression modes and received following results:
                Bit – 14.34%        
                Byte – 23.58%
                Compression - 3.3%
As an experiment, I introduced partial flush (PF) event in elements grammars, as I described in the original message. In my test application I called PF event every 100 messages or when interval between events (every event had timestamp) was more than 5 min (see weblog.xml). Results were:
                Bit – 14.36%
                Compression – 9.08%
Byte alignment is not a best solution for streamed XML. Personally, I would choose bit alignment or compression with flush event. It’s 40%+ better then straight byte alignment mode.
Yuri Delendik

----- Original Message ----
From: Daniel Peintner <daniel.peintner@gmail.com>
To: Yuri Delendik <yury_exi@yahoo.com>; public-exi@w3.org
Sent: Wednesday, October 1, 2008 1:11:59 PM
Subject: Re: [EXI] Partial flush

Hi Yuri,

> By buffer I meant 1 byte (8 bits). Now there is no way for EXI encoder send incomplete byte to the underline channel.

Sorry for my initial misunderstanding.

> Let's use XMPP example. XML stream data will be send in following manner without closing the underline channel: XML, {time delay}, XML, {time delay}, …, XML.
> For example:
> <stream>
> {some time delay}
> <stanza1>…</stanza1>
> {some time delay}
> <stanza2>…</stanza2>
> <stanza1>…</stanza1>
> {some time delay}
> ….
> </stream>
> And EXI encoder may hold bits after <stream> or </stanza1> and not send them to the channel, but other party is expecting that data as soon as possible -- those stanzas can be time critical material.

Now I see your problem.

A solution with EXI that could solve your problem may be the
byte-alignment option.
Byte-alignment generally results in EXI streams of larger sizes
compared with their bit-packed equivalents but byte-alignment provides
encoded data on aligned boundaries.

Please let us now whether this proposal is a possible solution for you.


-- Daniel

> ZLIB has partial flush functionality that aligns bits to byte border and sends compressed data to the channel, and makes it available to other party. But it does not terminate compression process.
> Compression's blockSize will not help is this case ether.
> Thank you.
> ----- Original Message ----
> From: Daniel Peintner <daniel.peintner@gmail.com>
> To: Yuri Delendik <yury_exi@yahoo.com>
> Cc: public-exi@w3.org
> Sent: Thursday, September 25, 2008 9:54:22 AM
> Subject: Re: [EXI] Partial flush
> Hi Yuri,
>> Some XML streaming protocols such as XMPP require data to be delivered as soon as possible.
>> When bit-packed option is selected, an EXI encoder may keep some data in its buffer for long
>> time before sending to it the channel.
> There is no reason to buffer data when EXI uses the bit-packed alignment option.
> EXI events and values can be streamed directly.
>> XMPP solves similar problem for ZLIB via partial data flush (http://www.xmpp.org/extensions
>> /xep-0138.html#impl). It will be nice to have a "flush" event that will align the stream to byte
>> border like self-contained (SC) event does. Same event may also restart compression, when a
>> compression option is selected.
> When EXI runs with either the compression or pre-compression alignment
> option turned on some kind of buffering is necessary. EXI re-orders
> data in a way that compression algorithms achieve higher compression
> ratios (see [1], Blocks and Channels).
> To allow devices with limited memory process large documents the
> blockSize option [2] is configurable. The blockSize value restricts
> the number of Attribute (AT) and Character (CH) values per block.
> Hope this answers your question,
> Regards,
> -- Daniel
> [1] http://www.w3.org/TR/exi/#compression
> [2] http://www.w3.org/TR/exi/#key-blockSizeOption
>> Thanks
Received on Thursday, 2 October 2008 01:32:29 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 19:52:43 UTC