Re: Data models and filtering (was: Re: Contacts API draft)

Hello,

This is an updated proposal on how to handle data sync and filtering,
based on Jonas' earlier suggestion, and discussions/alignment with
Chris. It satisfies the 2 main architectures which so far seem to be
target of sysapps:
- with JS side implementation of a database (IndexedDB), sync'd from
native data sources like SMS storage, focused purely on web apps (e.g.
Mozilla interested in this)
- native middleware and DB with data exposed to JS-side data models
(or 2-way sync'd IndexedDB and native DB), supporting both native and
web apps (e.g. Intel is interested in both this hybrid and the pure JS
model).

So, here is Jonas' proposal for reference:

> On Fri, Nov 9, 2012 at 10:41 PM, Jonas Sicking <jonas@sicking.cc> wrote:
>> A better solution might be to introduce the ability for an app to say
>> "give me a list of all changes that happened since the last time I
>> asked". That way the application can on first startup scan through the
>> contacts database and cache any information that it wants, it would
>> then say "please start recording changes for me". On next startup it
>> would simply get the list of changes since last run and update its
>> cache based on that. The API implementation would then queue any
>> modifications that are made and only drop those modifications once all
>> applications that have said "please record" has received the record.
>>
>> To make it more concrete, it could look something like this:
>>
>> interface ContactsManager {
>>   ...
>>   startTrackingChanges();
>>   ContactRequest getNewChanges();
>>   attribute EventHandler onchangesavailable;
>>   stopTrackingChanges(); // Not strictly needed, but seems like a good idea
>>   ...
>> };

This could be made a generic interface like the one presented next.
Before going on, I list some differences like:
- if the client wants to stop listening to changes, remove the event
listener on the EvenTarget interface
- if the client wants to (re)start listening to changes, call first
clearUpdates() before calling pullUpdates()
- otherwise a call to pullUpdates() will retrieve all changes since
the last call (or, if it's the first call, then the whole DB)
- a client can push the changes done to the native side (removals,
additions and modifications, in this order, and possibly in one
transaction)
- a client can low-pass filter the the sync updates, by setting a
minimum update interval, which if set e.g. to 100.0 ms, means the
notifications will be fired at least 100 ms apart. This can save
battery life in many applications.
When judging the design, please think the use cases through both
architectural contexts (hybrid and JS-only).

<code>
interface DataSync : EventTarget {
     /** notification event, fired when new updates are available
        * and next possible time it can be fired
        */
     attribute EventHandler onupdatesavailable;

     /** minimum interval when update notification events can be sent
       * this saves unnecessary wake-up's
       * measured in milliseconds, 0 meaning no constraints
       */
     attribute unsigned double  minUpdateInterval;

     /** manually pull items changed from last sync
       * PendingRequest.result will be a DataSyncItems object
       * optional filter: for big data usually only filtered data is needed
       */
     PendingRequest pullUpdates(optional AbstractFilter filter,
                                optional FilterOptions filterOptions);

     /** manually push items changed, it can be one transaction */
     PendingRequest pushUpdates(DataSyncItems items);

     /** reset sync window, start over with counting changes */
     PendingRequest clearUpdates();
};

/** data updates; first, removed id's are sync'd,
  * then added items, then changed items */
interface DataSyncUpdates {
    /** array of removed id's */
    attribute DOMString[] removedIds;

    /** array of items */
    attribute any              addedItems;

    /** array of items */
    attribute any              modifiedItems;
};

interface PendingRequest : EventTarget {
    readonly attribute DOMString readyState;
    readonly attribute any       result;
    readonly attribute DOMError  error;
    attribute EventHandler onsuccess;
    attribute EventHandler onerror;
};
</code>

This is used currently for the Messaging draft, and can also be used
for Contacts (and CallHistory), in every interface that needs to keep
track of data changes, be it messages, or message labels, or services.

It strongly encourages using system-wide id generation (e.g. id's
prefixed by service). These id's are not necessarily the DB primary
keys generated by the databases.

Then, here is a generic search interface:

<code>
interface DataSearch {
     PendingRequest find(optional AbstractFilter filter,
                         optional FilterOptions filterOptions);
};
</code>

The PendingRequest.results will contain the result set of the query.

Additionally, here is a generic interface for filtering from Chris,
used in Contacts, Messaging and CallHistory, which can be simple
filter (including ones from other specs) or a complex composite
filter.

<code>
[NoInterfaceObject]
interface AbstractFilter {
};
</code>

Note that it can use e.g. the SMSFilter too, from the SMS API:

<code>
[NoInterfaceObject]
interface SMSFilter : AbstractFilter {
             attribute Date?        startDate;
             attribute Date?        endDate;
             attribute DOMString[]? numbers;
             attribute DOMString?   delivery;
             attribute boolean?     read;
};
</code>

Or, it could be more complex and more generic filter like the ones
defined in Contacts API.

<code>
enum CompositeFilterOperator { "and", "or" };

Constructor(DOMString operator, AbstractFilter[] filters)]
interface CompositeFilter : AbstractFilter {
    attribute CompositeFilterOperator operator;
    attribute AbstractFilter[] filters;
};

enum AttributeFilterOperator { "is", "contains", "startswith",
"endswith", "exists" };

/* timestamp, to, cc, bcc, from, status, conversationId, labels,
folder, read, hasAttachment */
Constructor(DOMString filterBy, DOMString attributeOp, optional any
filterValue)]
interface AttributeFilter : AbstractFilter {
    attribute DOMString filterBy;
    attribute AttributeFilterOperator operator;
    attribute any filterValue;
};

/* timestamp */
Constructor(DOMString attributeName, optional any startValue, optional
any endValue)]
interface AttributeRangeFilter : AbstractFilter {
    attribute DOMString filterBy;
    attribute any startValue;
    attribute any endValue;
};

enum SortOrder { "ascending", "descending" };

dictionary FilterOptions {
    DOMString sortBy;
    SortOrder sortOrder = "ascending";
    unsigned long limit;
};
</code>

This permits search on any attribute of the object on which is
applied. It is possible to implement support for only a subset of the
attributes (documented, and with error returned on not supported
ones), but it is even simpler to just define a simple filter and use
that as AbstractFilter (with or without filter options).

When combined, these can satisfy use cases like:
- retrieving contacts lists,
- filtered transient (not saved to DB) contact lists ordered on
presence, with live update (e.g. chatroom roster)
- updating contacts from JS side (removed, added, modified) in a
single (or multiple) transaction(s)
- listing [messaging, call, etc] services and keeping them updated
(enabled/disabled, etc)
- retrieving messages, conversations, labels, folders and any filtered
subsets of these
- call history timeline, and conversation view
- mixed call and messaging conversation history
- etc, enabling a lot of creative designs.

The nice part is that the interface can also adopt simple optimized
filters when the platform implementation uses e.g. key-value store
instead of SQL or RDF database.

Also, the implementations have the choice to optimize DB access on the
native side, or on JS side, or even both.

Soon our Contacts and Messaging proposal will be updated for using
this (our Messaging draft is yet to be published).

Any suggestions or comments are appreciated.

Best regards,
Zoltan

Received on Friday, 16 November 2012 13:03:55 UTC