Fwd: ldp wishlist for crosscloud

FYI: Sandro just posted an excellent, detailed, wish list to the LDP WG.
Perhaps 75%+ could be related to RWW standards.

Would love to see any, or all of these points, move closer to
standardization...

---------- Forwarded message ----------
From: Sandro Hawke <sandro@w3.org>
Date: 9 November 2014 18:06
Subject: ldp wishlist for crosscloud
To: Linked Data Platform WG <public-ldp-wg@w3.org>


As you may know, these days most of my time is no longer W3C-staff but is
funded research toward building "Crosscloud", an architecture for software
which allows users to control their data and which should encourage
innovation by making it much easier to develop powerful, open multi-user
(social) software.

Back in January, we started off building on LDP, with Andrei creating
cimba.co.  It's a microblogging app, intending to replicate some of early
Twitter, in a completely decentralized way using generic
(non-application-specific) LDP.    To make that work, we had to extend LDP
with Access Control; clients can tell the server who can do what with each
research.   We also made no use of Direct Containers, Indirect Containers,
or Paging.   It's just Basic Containers, Access Control, WebID-TLS for
client authentication, Turtle for data, and non-RDF resources for photos.
   (Maybe I'm forgetting some details; for demo see 2 minute video at [1].)

While cimba basically works, it's painful in various ways and unable to do
many things, showing us that we need much more support from the servers.
We've also started building several more apps which are showing other
things that are important to have.

We don't have it all figured out yet, let alone implemented, but here are a
few of the thing we probably need.   I'm providing this list to help with
re-chartering, although most of these are not yet mature enough for
standardization.   Maybe they will be in 6-12 months, though.    As you
look at this list, one thing to figure out is how will we know when this
module is ready for the WG to take up.

== 1.  Queries

This is a big one.   It's impractical to have the cimba WebApp, running in
the browser, do all the GETS (hundreds, at least) every time it starts.  It
needs to do a small number of queries, and have the server manage all the
aggregation.   The server has to be able to query across other servers as
well as itself.

We're currently playing with forms of "Link-Following SPARQL", but also
with a more restricted MongoDB-like query language, both for easier
implementation and for response-time/load guarantees.

Queries make resource-paging obsolete, which is why I've lost interest in
paging.

== 2.  Change Notification to Web Servers

If a server acting on behalf of the end-user is going to aggregate data
from other servers, it needs to be able to keep its copy in sync.
Traditional web cache + polling works only when it's okay to be seconds or
minutes out of date; many multi-user apps require much more responsiveness
than that, so we see a need for one server to be able to subscribe to
change notification from another.

One might want something like PATCH to make this more efficient, but at the
moment it looks like we can keep the resources small enough that it doesn't
matter.

== 3.  Change Notification to Web Clients

Similarly, Web Apps often need to know immediately when data has changed.
 While it might be nice to have this be the same protocol as (2), our
preliminary investigation suggests the engineering trade-offs make that
impractical.   So, this needs to be its own protocol. Probably it's just a
tweak to the query protocol where query results, rather than being a single
response collecting all the results, are ongoing add-result and
remove-result events.

== 4.  Operation over WebSockets

It almost certainly makes sense to use WebSockets for (3), but it also
makes sense to use them for all the current LDP operations for high
performance.     A modest client and server can probably process at least
1000 GETs per second, but in practice, without WebSockets, they'll be
slowed an order of magnitude because of round trip delays.    That is, say
RTT is 50ms, so we can do 20 round trips per second.    Most browsers allow
at most 6 connections per hostname [2], so that's 120 round trips per
second, max, no matter how much CPU and RAM and bandwidth you have.

I'm still thinking about what this might look like.     Strawman is
something like each client-to-server message is a JSON object like {
"verb": "GET", "resource":"http://example.org", "accept": "text/html",
"seq":7 } and response are like { "in-reponse-to": 7, "status": 200,
"contentType": "text/html", "content": "<html>......</html>" }

So the higher levels don't have to know it's not normal HTTP, *but* we can
have hundreds or thousands of requests pipelined.     Also, we can have
multiple responses, or something, for event notification.   This would also
allow for more transactional operation, if desired.   (Maybe
"partial-response-to" and "final-response-to".)

== 5.  Non-Listing Containers

I want end-points that I can POST to, and GET some information about,
without being swamped by an enumeration of everything posted there.   I
don't want to have to include a Prefer header to avoid that swamping.

You might consider this a taste, but I think it's an important usability
issue.

Again, with querying, you probably don't want to just be dumping the list
of contained resources.   Querying also lets us control inlining, etc.
 Basically, if querying is available, I think we can skip serializing
membership/containment triples.

== 6.  PUT-to-Create

There are situations where the client needs to lay out, on the server, an
assortment of resources with carefully controlled URLs, such as a static
website with interlinked html, css, js, images, etc.    This should be
doable with PUT, where PUT creates the resource inside the container that
owns that URL space.

== 7.  DELETE WHERE

One of our current demo apps is a game that is likely to generate a dozen
resources per second per user.   Asking for each of those resources to be
individually deleted afterwards seems rather silly, even problematic, so a
DELETE WHERE operation would be nice.

Yes, one could put them all in a container in this case, and define it as a
kind of container that deletes its contained resources when it's deleted,,
but there are situations where that wont work as well.  Maybe we want to
delete the resources after about 60 seconds have gone by, for example.
 Easy to do with a DELETE WHERE, hard to do otherwise.

==  8.  WebMention for Data, backlinks used in Queries

The basics of WebMention are in-scope for the Social Web WG, but it's not
clear they'll apply it to arbitrary raw data, or say how the back-links are
made available for use in queries.   Like many of these, this might be
joint work with SWWG.

==  9.  Client Authentication

Arguable this is quite out of scope, and yet it's hard to operate without
it.   Especially things like (2) are easier with some kind of
authentication.

For a strawman of how easy it could be: https://github.com/sandhawke/
spot/blob/master/spec.md

== 10.  Access Control

Obviously.

My current radical theory is I only need is a flag that a page is
owner-only, public, or group-read, and then a way to define the group of
identities (see (9)) who can read it.    Most people imagine we need to
control a lot more than read access, and perhaps we do, but I'm currently
working with the theory that everyone makes their own contributions in
their own space, notifying but never actually "writing" to anyone else's.

== 11.  Combined Metadata and Content operations

I don't think I can put this very crisply, but I've started thinking about
resources as looking like this:

{ property1: value1,
   property2: value2,
   ...
   content: "<html>....</html>",
   contentType: "text/html"
   ...
}

and it's so much nicer.   Basically, every resource is properties-value
pairs, and some of that pv data is "content".    If you don't do something
like this, queries and notifications and all that require us to bifurcate
into a mechanism that's all about the content and another that's all about
the metadata.

LDP-RS's then become content-free resources, or null-content resources, but
much less fundamentally different.   With the current LDP framing, what
happens when you PUT an image to an LDP-RS or PUT rdf to what you created
as an image?   This model clears that up nicely.

But this might only work in the face of other assumptions I'm making, like
the only triples at <R> are in a graph rooted at <R>, so you can think of
them all as properties of R.    Also I've resolved httpRange-14 by saying
I'm only interested in proper information-resource-denoting URLs, and you
can use indirect properties for talking about people, places, events, etc.
  Maybe those radical assumptions are necessary for making this work.

12.  Forwarding

We need to be able to move resources, because it's very hard to pick a URL
and stick to it for decades.   And if it's used as part of other apps, and
you don't stick to it, you'll break them.   The fear of this will, I
suspect, significantly impede adoption.

I propose three mechanisms.   Any one of them might work; between the three
I'm fairly confident.

1.  Servers SHOULD check all their outgoing links at least once every 30
days.   If they get a 301 response, they SHOULD update the link in place.
 Valid reason not to change it is this is some kind of a frozen/static page
that can't be changed.

2.  When a client gets a 301, following a link it got from server A, it
should notify server A, so A can rewrite the link sooner.   This could use
a .well-known end-point on A, or there could be a Report-Link-Issues-To
header on every resource which A serves telling clients how to report any
301s (and 404s) it finds.

3.  The notification mechanism (2) above, should include move
notifications, so when a page is being watched, if it moves the watcher
will be immediately notified and able to change its link.

All this works much better if in addition to 301 we have a way to say a
whole tree has moved.    That is, all URLs starting http://foo.example/x/
should not be considered redirected to http://bar.example/y/, etc.

With these mechanisms in place, links from compliant servers should start
to transition quickly and drop off to zero after 30 days. Obviously links
from hand-maintained resources, and printed on paper, etc, wont change, but
those are usually consumed by humans who are better able to deal with a
broken link anyway.

== More...

I'm sure there's more, but this gives the general shape of things. Do we
want the new charter to target some of these?   To allow for some of
these?   And again: how do we assess when each of these is mature enough
for a WG to begin looking at it?

Thanks for considering this.

      -- Sandro


[1] https://www.youtube.com/watch?v=z0_XaJ97rF0
[2] http://www.browserscope.org/?category=network&v=top