RE: [EXI] Partial flush

Republishing to the list for everyone's benefit. Yuri and Daniel received my
original message, but it was not posted to the list due to an unknown "from"
address. (Need to fix my iPhone config :-).  

---------------------

Yuri,

Try byte aligned EXI followed by generic zip with partial flush. I think
this will likely be your best solution for partial flush applications and
does not require the added complexity of adding partial flush events to the
EXI format.

   Cheers,

   John

Sent from my iPhone 3G 

> -----Original Message-----
> From: public-exi-request@w3.org 
> [mailto:public-exi-request@w3.org] On Behalf Of Yuri Delendik
> Sent: Wednesday, October 01, 2008 6:32 PM
> To: Daniel Peintner; public-exi@w3.org
> Subject: Re: [EXI] Partial flush
> 
> 
> Hi Daniel,
>  
> Thank you for suggestion. As EXI user I would expect maximum 
> compression and byte-alignment does not provide that.
>  
> I ran EXI compression on weblog.xml (2,648,181 bytes) from 
EXI testing framework data using bit and byte alignment, and > compression
modes and received following results:
>                 Bit – 14.34%
>                 Byte – 23.58%
>                 Compression - 3.3%
>  
> As an experiment, I introduced partial flush (PF) event in 
> elements grammars, as I described in the original message. In 
> my test application I called PF event every 100 messages or 
> when interval between events (every event had timestamp) was 
> more than 5 min (see weblog.xml). Results were:
>                 Bit – 14.36%
>                 Compression – 9.08%
>  
> Byte alignment is not a best solution for streamed XML. 
> Personally, I would choose bit alignment or compression with 
> flush event. It’s 40%+ better then straight byte alignment mode.
>  
> Regards,
> Yuri Delendik
> 
> 
> 
> ----- Original Message ----
> From: Daniel Peintner <daniel.peintner@gmail.com>
> To: Yuri Delendik <yury_exi@yahoo.com>; public-exi@w3.org
> Sent: Wednesday, October 1, 2008 1:11:59 PM
> Subject: Re: [EXI] Partial flush
> 
> 
> Hi Yuri,
> 
> > By buffer I meant 1 byte (8 bits). Now there is no way for 
> EXI encoder send incomplete byte to the underline channel.
> 
> Sorry for my initial misunderstanding.
> 
> > Let's use XMPP example. XML stream data will be send in 
> following manner without closing the underline channel: XML, 
> {time delay}, XML, {time delay}, …, XML.
> >
> > For example:
> > <stream>
> > {some time delay}
> > <stanza1>…</stanza1>
> > {some time delay}
> > <stanza2>…</stanza2>
> > <stanza1>…</stanza1>
> > {some time delay}
> > ….
> > </stream>
> >
> > And EXI encoder may hold bits after <stream> or </stanza1> 
> and not send them to the channel, but other party is 
> expecting that data as soon as possible -- those stanzas can 
> be time critical material.
> 
> Now I see your problem.
> 
> A solution with EXI that could solve your problem may be the 
> byte-alignment option.
> Byte-alignment generally results in EXI streams of larger 
> sizes compared with their bit-packed equivalents but 
> byte-alignment provides encoded data on aligned boundaries.
> 
> Please let us now whether this proposal is a possible 
> solution for you.
> 
> Thanks,
> 
> -- Daniel
> 
> 
> 
> 
> >
> > ZLIB has partial flush functionality that aligns bits to 
> byte border and sends compressed data to the channel, and 
> makes it available to other party. But it does not terminate 
> compression process.
> >
> > Compression's blockSize will not help is this case ether.
> >
> > Thank you.
> >
> > ----- Original Message ----
> > From: Daniel Peintner <daniel.peintner@gmail.com>
> > To: Yuri Delendik <yury_exi@yahoo.com>
> > Cc: public-exi@w3.org
> > Sent: Thursday, September 25, 2008 9:54:22 AM
> > Subject: Re: [EXI] Partial flush
> >
> >
> > Hi Yuri,
> >
> >> Some XML streaming protocols such as XMPP require data to 
> be delivered as soon as possible.
> >> When bit-packed option is selected, an EXI encoder may 
> keep some data 
> >> in its buffer for long time before sending to it the channel.
> >
> > There is no reason to buffer data when EXI uses the 
> bit-packed alignment option.
> > EXI events and values can be streamed directly.
> >
> >> XMPP solves similar problem for ZLIB via partial data flush 
> >> (http://www.xmpp.org/extensions /xep-0138.html#impl). It 
> will be nice 
> >> to have a "flush" event that will align the stream to byte border 
> >> like self-contained (SC) event does. Same event may also 
> restart compression, when a compression option is selected.
> >
> > When EXI runs with either the compression or 
> pre-compression alignment 
> > option turned on some kind of buffering is necessary. EXI re-orders 
> > data in a way that compression algorithms achieve higher 
> compression 
> > ratios (see [1], Blocks and Channels).
> >
> > To allow devices with limited memory process large documents the 
> > blockSize option [2] is configurable. The blockSize value restricts 
> > the number of Attribute (AT) and Character (CH) values per block.
> >
> > Hope this answers your question,
> >
> > Regards,
> >
> > -- Daniel
> >
> > [1] http://www.w3.org/TR/exi/#compression
> > [2] http://www.w3.org/TR/exi/#key-blockSizeOption
> >
> >
> >
> >> Thanks
> >>
> >>
> >
> 
>

Received on Wednesday, 8 October 2008 18:12:36 UTC