RE: Data models and filtering (was: Re: Contacts API draft)

Hi Zoltan,

This looks very good :) The AbstractFilter and PendingRequest interfaces would be very useful beyond the Contacts / Messaging / Telephony areas, for instance for media browsing / filtering.

The PendingRequest interface now looks very much like a Javascript "Promise" pattern, maybe it would make sense to add a "then" method to the PendingRequest that sets the onsuccess and onerror attributes, so for instance one could perform a search like this:

myDataSearch.find(myFilter).then(onSuccess, onError)


Regards,

Luc

-----Original Message-----
From: Kis, Zoltan [mailto:zoltan.kis@intel.com] 
Sent: Friday, November 16, 2012 2:03 PM
To: Jonas Sicking
Cc: Dumez, Christophe; Tantek Çelik; EDUARDO FULLEA CARRERA; public-sysapps@w3.org; Carr, Wayne; Poussa, Sakari; JOSE MANUEL CANTERA FONSECA
Subject: Re: Data models and filtering (was: Re: Contacts API draft)

Hello,

This is an updated proposal on how to handle data sync and filtering, based on Jonas' earlier suggestion, and discussions/alignment with Chris. It satisfies the 2 main architectures which so far seem to be target of sysapps:
- with JS side implementation of a database (IndexedDB), sync'd from native data sources like SMS storage, focused purely on web apps (e.g.
Mozilla interested in this)
- native middleware and DB with data exposed to JS-side data models (or 2-way sync'd IndexedDB and native DB), supporting both native and web apps (e.g. Intel is interested in both this hybrid and the pure JS model).

So, here is Jonas' proposal for reference:

> On Fri, Nov 9, 2012 at 10:41 PM, Jonas Sicking <jonas@sicking.cc> wrote:
>> A better solution might be to introduce the ability for an app to say 
>> "give me a list of all changes that happened since the last time I 
>> asked". That way the application can on first startup scan through 
>> the contacts database and cache any information that it wants, it 
>> would then say "please start recording changes for me". On next 
>> startup it would simply get the list of changes since last run and 
>> update its cache based on that. The API implementation would then 
>> queue any modifications that are made and only drop those 
>> modifications once all applications that have said "please record" has received the record.
>>
>> To make it more concrete, it could look something like this:
>>
>> interface ContactsManager {
>>   ...
>>   startTrackingChanges();
>>   ContactRequest getNewChanges();
>>   attribute EventHandler onchangesavailable;
>>   stopTrackingChanges(); // Not strictly needed, but seems like a good idea
>>   ...
>> };

This could be made a generic interface like the one presented next.
Before going on, I list some differences like:
- if the client wants to stop listening to changes, remove the event listener on the EvenTarget interface
- if the client wants to (re)start listening to changes, call first
clearUpdates() before calling pullUpdates()
- otherwise a call to pullUpdates() will retrieve all changes since the last call (or, if it's the first call, then the whole DB)
- a client can push the changes done to the native side (removals, additions and modifications, in this order, and possibly in one
transaction)
- a client can low-pass filter the the sync updates, by setting a minimum update interval, which if set e.g. to 100.0 ms, means the notifications will be fired at least 100 ms apart. This can save battery life in many applications.
When judging the design, please think the use cases through both architectural contexts (hybrid and JS-only).

<code>
interface DataSync : EventTarget {
     /** notification event, fired when new updates are available
        * and next possible time it can be fired
        */
     attribute EventHandler onupdatesavailable;

     /** minimum interval when update notification events can be sent
       * this saves unnecessary wake-up's
       * measured in milliseconds, 0 meaning no constraints
       */
     attribute unsigned double  minUpdateInterval;

     /** manually pull items changed from last sync
       * PendingRequest.result will be a DataSyncItems object
       * optional filter: for big data usually only filtered data is needed
       */
     PendingRequest pullUpdates(optional AbstractFilter filter,
                                optional FilterOptions filterOptions);

     /** manually push items changed, it can be one transaction */
     PendingRequest pushUpdates(DataSyncItems items);

     /** reset sync window, start over with counting changes */
     PendingRequest clearUpdates();
};

/** data updates; first, removed id's are sync'd,
  * then added items, then changed items */ interface DataSyncUpdates {
    /** array of removed id's */
    attribute DOMString[] removedIds;

    /** array of items */
    attribute any              addedItems;

    /** array of items */
    attribute any              modifiedItems;
};

interface PendingRequest : EventTarget {
    readonly attribute DOMString readyState;
    readonly attribute any       result;
    readonly attribute DOMError  error;
    attribute EventHandler onsuccess;
    attribute EventHandler onerror;
};
</code>

This is used currently for the Messaging draft, and can also be used for Contacts (and CallHistory), in every interface that needs to keep track of data changes, be it messages, or message labels, or services.

It strongly encourages using system-wide id generation (e.g. id's prefixed by service). These id's are not necessarily the DB primary keys generated by the databases.

Then, here is a generic search interface:

<code>
interface DataSearch {
     PendingRequest find(optional AbstractFilter filter,
                         optional FilterOptions filterOptions); }; </code>

The PendingRequest.results will contain the result set of the query.

Additionally, here is a generic interface for filtering from Chris, used in Contacts, Messaging and CallHistory, which can be simple filter (including ones from other specs) or a complex composite filter.

<code>
[NoInterfaceObject]
interface AbstractFilter {
};
</code>

Note that it can use e.g. the SMSFilter too, from the SMS API:

<code>
[NoInterfaceObject]
interface SMSFilter : AbstractFilter {
             attribute Date?        startDate;
             attribute Date?        endDate;
             attribute DOMString[]? numbers;
             attribute DOMString?   delivery;
             attribute boolean?     read;
};
</code>

Or, it could be more complex and more generic filter like the ones defined in Contacts API.

<code>
enum CompositeFilterOperator { "and", "or" };

Constructor(DOMString operator, AbstractFilter[] filters)] interface CompositeFilter : AbstractFilter {
    attribute CompositeFilterOperator operator;
    attribute AbstractFilter[] filters;
};

enum AttributeFilterOperator { "is", "contains", "startswith", "endswith", "exists" };

/* timestamp, to, cc, bcc, from, status, conversationId, labels, folder, read, hasAttachment */ Constructor(DOMString filterBy, DOMString attributeOp, optional any filterValue)] interface AttributeFilter : AbstractFilter {
    attribute DOMString filterBy;
    attribute AttributeFilterOperator operator;
    attribute any filterValue;
};

/* timestamp */
Constructor(DOMString attributeName, optional any startValue, optional any endValue)] interface AttributeRangeFilter : AbstractFilter {
    attribute DOMString filterBy;
    attribute any startValue;
    attribute any endValue;
};

enum SortOrder { "ascending", "descending" };

dictionary FilterOptions {
    DOMString sortBy;
    SortOrder sortOrder = "ascending";
    unsigned long limit;
};
</code>

This permits search on any attribute of the object on which is applied. It is possible to implement support for only a subset of the attributes (documented, and with error returned on not supported ones), but it is even simpler to just define a simple filter and use that as AbstractFilter (with or without filter options).

When combined, these can satisfy use cases like:
- retrieving contacts lists,
- filtered transient (not saved to DB) contact lists ordered on presence, with live update (e.g. chatroom roster)
- updating contacts from JS side (removed, added, modified) in a single (or multiple) transaction(s)
- listing [messaging, call, etc] services and keeping them updated (enabled/disabled, etc)
- retrieving messages, conversations, labels, folders and any filtered subsets of these
- call history timeline, and conversation view
- mixed call and messaging conversation history
- etc, enabling a lot of creative designs.

The nice part is that the interface can also adopt simple optimized filters when the platform implementation uses e.g. key-value store instead of SQL or RDF database.

Also, the implementations have the choice to optimize DB access on the native side, or on JS side, or even both.

Soon our Contacts and Messaging proposal will be updated for using this (our Messaging draft is yet to be published).

Any suggestions or comments are appreciated.

Best regards,
Zoltan

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Received on Monday, 19 November 2012 09:05:54 UTC