RE: Proposal for including EXI in XMPP

Hi Peter,

I'd like to propose to collate the set of options to make it more
amenable to EXI applications.

- Alignment Type

  One of compress, preCompress, byteAligned, bitPacked.
  It is hard to imagine a server that does not support bitPacked.

- Grammar Options

  Default grammar with or without each of NS, CM, PI, SC, DTD
    or
  Strict grammar

  where we can assign Strict grammar of value 0, and
  Default grammar with value 1. We then can treat NS, CM, PI, SC, DTD as
  bit flags. For example, NS=10, CM=100, PI=1000, SC=10000 and DTD=100000.
  Flags can be combined with default grammar (i.e. 1).

  When the client and server negotiates the grammar option, each
  iteration decreases the value (i.e. remove a flag) to eventually
  converge to 0 (i.e. strict).

- Lexical Preservation

  Apart from the issue of which setting is more difficult, I think
  all server should support the value of "false".

- datatypeRepresentationMap

  (Support of this option is optional in EXI.)

- blockSize

  The default blockSize of 1,000,000 may be a bit too large to require all
  servers to support. The server and client can always find the lesser of
  the two.

- valueMaxLength

  We should let the parties to find the least value.

- valuePartitionCapacity

  We should let the parties to find the least value.


Regards,

taki



From: Peter Waher [mailto:Peter.Waher@clayster.com]
Sent: Tuesday, March 19, 2013 3:07 PM
To: Takuki Kamiya
Cc: FABLET Youenn (Youenn.Fablet@crf.canon.fr); Joachim Lindborg; John Schneider; mact-usa@att.net; Peter Saint-Andre; public-exi@w3.org; Stephen Williams <sdw@lig.net> (sdw@lig.net); XMPP Standards; Yusuke DOI
Subject: RE: Proposal for including EXI in XMPP

Hello Takuki

Thank you for your quick and instructive reply.

Before adding this to the proposal, I would like to hear from you, and from others interested in this theme:

What EXI options should we require the XMPP server to support (MUST)? If we could agree on a set of options the server must implement we can make the negotiation simpler, and clients can assume these options to be supported.

Regarding the Boolean and enumeration options: Since these normally default to "false" if not specified (and alignment to bit-packed), can we require these options to be supported of all XMPP Servers implementing this XEP? Or can anyone foresee a case where these options cannot/should not be implemented?

Sincerely,
Peter Waher


From: Takuki Kamiya [mailto:tkamiya@us.fujitsu.com]
Sent: den 19 mars 2013 18:43
To: Peter Waher
Cc: FABLET Youenn (Youenn.Fablet@crf.canon.fr<mailto:Youenn.Fablet@crf.canon.fr>); Joachim Lindborg; John Schneider; mact-usa@att.net<mailto:mact-usa@att.net>; Peter Saint-Andre; public-exi@w3.org<mailto:public-exi@w3.org>; Stephen Williams <sdw@lig.net<mailto:sdw@lig.net>> (sdw@lig.net<mailto:sdw@lig.net>); XMPP Standards; Yusuke DOI
Subject: RE: Proposal for including EXI in XMPP

Hi Peter,

The default option value defined for EXI Options document when a field
was omitted is chosen based more on the observation that such a setting
would be more common in the use of EXI. There are natural correspondences
between the easiness and the commonality of the option values in many cases.

However, there are some cases where this observation does not hold.

Lexical preservation is in a sense easier than the default lexical
*non*-preservation because you do not need to implement typed
representations when lexical preservation is on. (Can we require the
server to always support both?)

Similarly, strict grammar may be deemed easier than the default grammar
setting because strict grammar takes less footprint and does not permit
adding things like CM or PI. Note strict grammar is *not* the default.

Having said that, these should be rather harmless. When the client asks
for strict grammar setting for example, it should be ok for the server to
respond with counter proposal of default grammar setting. The client
may be a bit puzzled with the answer (because it thinks "default" grammar
is harder), it can simply decide not to continue.

For the alignment settings (i.e. bit-packed, byte-aligned, pre-compression
and compression), we can take it for granted that compression is harder
than pre-compression, and pre-compression is harder than both byte-aligned
and bit-packed. But the order is equivocal between bit-packed, byte-aligned.

If we can require the server to always support both bit-packed and byte-aligned,
then the issue would go away. The server just need to take care to downgrade
compression to either pre-compression or byte-aligned, and pre-compression
to byte-aligned. If the server downgrade compression to bit-packed for example,
that is not really a downgrade, and the client may be baffled at that point if
it does not support bit-packed (but supports byte-aligned).

It seems that the server can only downgrade according to a lineage of
well-defined difficulty, whereas the client should be able make a counter
proposal using an alternate setting outside the lineage when one lineage
did not turn to work out.

For numeric values, the lineage relationship is apparent. The larger value
means more difficulty. Most binary values have lineages where the value
true indicates more difficulty than the value false. Lexical preservation
case is a binary case where there is no lineage of difficulty, and the
strict grammar case is a binary case where the lineage is opposite
(i.e. true case is easier). The alignment case is the one that has only
a partial lineage.

Regards,

taki




From: Peter Waher [mailto:Peter.Waher@clayster.com]
Sent: Tuesday, March 19, 2013 11:58 AM
To: Takuki Kamiya
Cc: FABLET Youenn (Youenn.Fablet@crf.canon.fr<mailto:Youenn.Fablet@crf.canon.fr>); Joachim Lindborg; John Schneider; mact-usa@att.net<mailto:mact-usa@att.net>; Peter Saint-Andre; public-exi@w3.org<mailto:public-exi@w3.org>; Stephen Williams <sdw@lig.net<mailto:sdw@lig.net>> (sdw@lig.net<mailto:sdw@lig.net>); XMPP Standards; Yusuke DOI
Subject: RE: Proposal for including EXI in XMPP

Hello Takuki

Thanks for your input.

I'm trying to see to which extent an ordering of enumeration or boolean options has any meaning.

My first observation is that the default values of the options must be supported by the server. They could be seen as "easier". Furthermore, they are all "false". So "false" < "true" in this case. This could be generalized in a MUST-statement, that the XMPP Server must support the default values of all EXI options for the EXI versions it is supporting.

Regarding the alignment attribute, it's more difficult to order something based on "difficulty", since the default value is "bit-packed", which is "harder" in a sense than byte aligned. Could this be resolved using a MUST-statement that the XMPP server must support all three alignments?

Sincerely,
Peter Waher

From: Takuki Kamiya [mailto:tkamiya@us.fujitsu.com]
Sent: den 18 mars 2013 16:03
To: Peter Waher
Cc: FABLET Youenn (Youenn.Fablet@crf.canon.fr<mailto:Youenn.Fablet@crf.canon.fr>); Joachim Lindborg; John Schneider; mact-usa@att.net<mailto:mact-usa@att.net>; Peter Saint-Andre; public-exi@w3.org<mailto:public-exi@w3.org>; Stephen Williams <sdw@lig.net<mailto:sdw@lig.net>> (sdw@lig.net<mailto:sdw@lig.net>); XMPP Standards; Yusuke DOI
Subject: RE: Proposal for including EXI in XMPP

Hi Peter,

The updated text in section 3.2 looks good. Thanks for incorporating the change.

Now, I have another aspect of the XEP that I am not very clear about.

It is about section 2.3 Proposing compression parameters.

Though I understand the described mechanics of parameter convergence,
I wonder if there were things that could improve the mechanism further.

For numeric parameters such as blockSize or valueMaxLength, each option value
can be deterministically ordered. On the other hand, the order among values of
options such as alignment option or lexical preservation option is subject to
judgment.

For those not-totally-ordered parameters at least, allowing for specifying
ordered preference values (instead of a single value) may help the negotiation
to succeed, avoiding unsuccessful marriages that could otherwise be successfully
matched.

Regards,

taki




From: Peter Waher [mailto:Peter.Waher@clayster.com]
Sent: Friday, March 15, 2013 5:57 PM
To: Takuki Kamiya
Cc: FABLET Youenn (Youenn.Fablet@crf.canon.fr<mailto:Youenn.Fablet@crf.canon.fr>); Joachim Lindborg; John Schneider; mact-usa@att.net<mailto:mact-usa@att.net>; Peter Saint-Andre; public-exi@w3.org<mailto:public-exi@w3.org>; Stephen Williams <sdw@lig.net<mailto:sdw@lig.net>> (sdw@lig.net<mailto:sdw@lig.net>); XMPP Standards; Yusuke DOI
Subject: RE: Proposal for including EXI in XMPP

Hello Taki

Thanks for your valuable input. You're of course correct, so I've corrected §3.2 accordingly. I've attached the latest version.

Sincerely,
Peter Waher


From: Takuki Kamiya [mailto:tkamiya@us.fujitsu.com]
Sent: den 15 mars 2013 21:26
To: Peter Waher
Cc: FABLET Youenn (Youenn.Fablet@crf.canon.fr<mailto:Youenn.Fablet@crf.canon.fr>); Joachim Lindborg; John Schneider; mact-usa@att.net<mailto:mact-usa@att.net>; Peter Saint-Andre; public-exi@w3.org<mailto:public-exi@w3.org>; Stephen Williams <sdw@lig.net<mailto:sdw@lig.net>> (sdw@lig.net<mailto:sdw@lig.net>); XMPP Standards; Yusuke DOI
Subject: RE: Proposal for including EXI in XMPP

Hi Peter,

Each EXI Body needs to always start with SD and ends with ED. SD is ethereal
(i.e. zero-bit), so its presence is indiscernible. On the other hand, ED often
carries bits (depending on EXI preservation settings in effect). Therefore,
stripping EXI Body grammar of them would amount to an alteration to the EXI
specification, which I think we should avoid. I suggest to adopt EXI body as a
whole.

Exerpted from the first paragraph of Section 3.2:
> The transmission of EXI-compressed stanzas takes the form of a sequence
> of EXI bodies. In order for the recipient to be able to correctly interpret
> these incoming EXI bodies, the sender is required to flush any pending bits
> at the end of the last End Element (EE) event for each stanza and then send
> any pending bytes available in the output buffer. Since this makes sure
> each EXI body starts at an even byte boundary, it permits the recipient to
> decompress the body into an XML stanza.

Assuming that each stanza is represented as an EXI body, it is the ED event
(instead of EE) for which flush operation needs to occur.

Regards,

taki


From: Peter Waher [mailto:Peter.Waher@clayster.com]
Sent: Friday, March 15, 2013 8:00 AM
To: Takuki Kamiya
Cc: FABLET Youenn (Youenn.Fablet@crf.canon.fr<mailto:Youenn.Fablet@crf.canon.fr>); Joachim Lindborg; John Schneider; mact-usa@att.net<mailto:mact-usa@att.net>; Peter Saint-Andre; public-exi@w3.org<mailto:public-exi@w3.org>; Stephen Williams <sdw@lig.net<mailto:sdw@lig.net>> (sdw@lig.net<mailto:sdw@lig.net>); XMPP Standards; Yusuke DOI
Subject: RE: Proposal for including EXI in XMPP

Hello Takuki,

Thank you for your valuable comments. I've rewritten §3.2 according to the ideas you presented. I also specify when the stream ends in the same section.

Would this address your comments?

I've attached the most recent revision.

Sincerely,
Peter Waher


From: Takuki Kamiya [mailto:tkamiya@us.fujitsu.com]
Sent: den 15 mars 2013 04:00
To: Peter Waher; public-exi@w3.org<mailto:public-exi@w3.org>
Cc: Joachim Lindborg (joachim.lindborg@sust.se<mailto:joachim.lindborg@sust.se>)
Subject: RE: Proposal for including EXI in XMPP

Hi Peter,

Thank you for sharing your XMPP work which already appear
to have collated many relevant points in producing an excellent
draft specification.

Section 3.2 in the updated version of the document describes
successive, back-to-back use of multiple EXI bodies. Since
this usage is something EXI does not directly describe, you might
want to make sure EXI bodies (except for the first EXI body
which begins immediately after a EXI header) each start at a byte
boundary by explicitly mentioning that is the case.

Also, you may want to use the terminology "EXI body" explicitly
in order to avoid each sequence of (SD ... ED) to be understood
as an EXI document. EXI 1.0 requires string table content to be
reset for each EXI document, which is in conflict with what I think
you want to achieve.

One thing that was not very clear to me was the way the recipient
recognizes that there is no more EXI body following one. Is there
going to be a stanza that represents it is the last one in the
communication?

Regards,

taki



From: Peter Waher [mailto:Peter.Waher@clayster.com]
Sent: Wednesday, March 13, 2013 9:43 AM
To: public-exi@w3.org<mailto:public-exi@w3.org>
Cc: Joachim Lindborg (joachim.lindborg@sust.se<mailto:joachim.lindborg@sust.se>)
Subject: Proposal for including EXI in XMPP

Hello

We have made a first draft of a proposal for incorporating EXI into XMPP networks. (See attached files.)

Anybody with an interest in EXI & XMPP are welcome to join us in this effort, please contact us. Any comments, suggestions, etc., on the contents of these documents are warmly welcomed and appreciated.

Sincerely,
Peter Waher