Re: Regarding user interfaces and smart clients

Hi Nathan

That's some interesting read. I will share some thoughts inline.

On 2015-03-12 15:10, Nathan Ridley wrote:
 > I'd really welcome any comments or thoughts to the following ideas that
 > I've been formulating. There's more generic linked data stuff in this
 > post than Hydra-specific discussion, but I'm including it here because
 > the entire crux of what I'm talking about revolves around the
 > construction of a modern web application using JSON-LD and Hydra.
 >
 > First, my assumptions based on what I've learned (please correct me on
 > anything I'm wrong about):
 >
 > 1) To play nicely with the ideals of the semantic web, my client should
 > be able to consume external linked data, and external clients should be
 > able to consume my data. The same goes for my Hydra API, which is of
 > course the entire point of Hydra.
 >
 > 2) To be purely HATEOAS/REST-compliant, I should be able to point my
 > client at any endpoint which offers data in a format I can support (in
 > my case, at least JSON-LD and Hydra) and have it be able to offer some
 > kind of user experience using only what is returned from that endpoint.
 > Any given request should include links so that a client can advertise
 > potential further actions to the user. The client should not make any
 > assumptions in advance about what will be returned, because those
 > assumptions will inevitably be wrong if anything about the endpoint
 > changes in the future.

I'm not entirely with you about the assumptions the client should or 
shouldn't make. Of course it is true for 100% generic clients but I 
think that below you mostly write about specialized clients. A 
specialized client would likely operate in a specific domain and a such 
you can expect that certain data is displayed differently and maybe some 
properties are ignored. It could use a generic view for unexpected data.

 >
 > ## User Interface ##
 >
 > My experience with application design is that the more generic you make
 > a client's capabilities, the harder it is to offer a really good user
 > experience. This of course is a problem when we're talking about linked
 > data and Hydra, as being generic is pretty much a necessity. So I've
 > been thinking about how we can "have our cake and eat it too". How can
 > we offer a great user experience, tailor our application to our
 > website's needs, and yet somehow remain HATEOAS/REST compliant so that
 > we remain completely interoperable with clients that know nothing about
 > our application?
 >
 > There are no two ways about this. User interface is really, really
 > important, and I can't help but notice a lack of focus on this fact by
 > the semantic web community. Unless you're talking about completely
 > autonomous agents, which I believe is Ruben's area of interest, you
 > simply can't get away from thinking about the user interface, and how
 > you might offer a good user experience. UI/UX experts get paid because
 > they spend the time to consider the data and service in advance and
 > craft something that the user is going to love, and which is going to do
 > the best job possible in allowing the user to interact with a service
 > and its data. This usually involves careful analysis of what the service
 > provides, what is important, and the various workflows that the user is
 > expected to engage in.

This is very important. From it's inception Semantic Web has mostly 
focused on autonomous or intelligent agents, while most of the web is 
designed on per-app basis. And for good reason as you point out. That 
may be the main reason why SW is largely ignored but the general public 
- it too often seems impractical and addresses issues that aren't 
paramount for most developers.

 >
 > How do we get our client to fill this role adequately? In all honestly,
 > I don't think we can, given what is on the table (of specifications) at
 > this point. I've seen discussions on the list where it is argued that
 > attributes such as labels and so forth have no place in the data because
 > these are presentational attributes and should not be intrinsically
 > linked to the data. I understand the argument, but then something still
 > needs to be offered whereby we can advertise metadata with which a
 > decent user interface can be constructed.

Well, the holy grail of SW would be that each property or type can be 
dereferenced so that a client can discover the metadata from relevant 
directly vocabulary. That is however impossible for very practical 
reasons: performance (many and potentially slow requests), unstable or 
un-dereferencable vocabularies and incomplete metadata (think 
translations and various lingustic quirks). Also one could argue that 
such metadata doesn't belong to the ontology/vocabulary itself anyway. 
In such case a given client has to know that information up-front 
anyway, much like it is done without any SW technologies. Whether a 
server publishes such metadata and how is another topic. I do however 
agree that this must not be part of Hydra. The problem is that 
introducing yet another vocabulary/scheme for publishing that along 
Hydra will only strengthen the perception that Semantic Web or Linked 
Data is difficult, because you need so many pieces: JSON-LD/RDF, Hydra, 
some metadata vocabulary.

 >
 > Questions which are precursors to an implementation model that I think
 > would work:
 >
 > (a) If I return a JSON-LD document in response to an HTTP request, the
 > general idea of hypermedia and linked data is that I should include
 > hypermedia controls to related data. Hypermedia controls seem to me be
 > context-specific metadata and not always intrinsically-related to the
 > requested resource. Is this line of thinking correct?

This one has baffled me. Consider a traditional web app, where on every 
page there is a login/logout link visible depending on the logon status. 
How does that correspond to a Linked Data resource? Do you include a 
relevant login/logout operation to every resource based on the request's 
authentication? I would assume you have to so that all possible state 
transitions are given with a response.

 >
 > (b) As above, is it generally understood that the document and its
 > related hypermedia controls have a dynamic relationship? For example, if
 > I return information about a user, then perhaps a link to the user's
 > account history will be included, or perhaps it won't; maybe the person
 > making the request doesn't have the appropriate credentials and
 > shouldn't be offered certain links.

Possibly but not necessarily. Again the HTML analogy. Consider pages 
which always include links to protected resources but redirect to login 
page when required. I think it would be reasonable to offer such 
links/operations in a representation and return 401 status with a log-in 
operation if the user is unauthenticated.

 >
 > (c) Building further on the above points; I know I also want to include
 > a link to a Hydra entry point, but is it appropriate to include a link
 > to that entry point from every document returned from my application?
 > One would assume that any route offered by my application may be called
 > as a first point of entry, and as such, the entry point would need to be
 > advertised so that the client is able to figure out what the API can do.

This is similar to what I described above. For example consider a menu 
displayed on every page. Do we offer such "static" links with every 
response? If not, how would you build a "real" page from such resource's 
representation, especially in a generic client? Unfortunately there are 
a number of similar issues. Think about hierarchies of resources. 
Usually pages include shared parts, which contain parent's data. For 
example a book chapter page displayed along with chapter listing and the 
book's title, etc.

 >
 > So my implementation model that has been materialising in my head looks
 > something like this:
 >
 > 1) All general requests to a server are subject to content negotiation
 > with respect to the data requested and the specified Accept header. So
 > in most cases, if the client requests JSON-LD, that's what it gets. If
 > it requests HTML though, we need to somehow decide how to render the
 > [arbitrary] data using HTML. That HTML would therefore need to include
 > enough so that the full client can bootstrap itself and then display the
 > data that was originally requested.

I'm not sure I understand the above.

 >
 > 2) A client cannot truly be intelligent. If it receives an object of
 > @type schema:LocalBusiness, it isn't going to know that one attribute is
 > more important than another, or that certain attributes probably don't
 > need to be displayed initially, or that certain groups of attributes, if
 > appearing in tandem, would be better rendered using a specialised view,
 > rather than individually in a table of values. We need to teach our
 > client how to render a user interface in a way that both suits our
 > application's domain-specific requirements, and can also adapt to linked
 > data that is beyond the application's initial scope.

By "teach" do you simply mean that specific page design/layout is 
prepared by developers for schema:LocalBusiness and unexpected data is 
handled in a generic way? I don't see a better way currently. Again, 
building more such "intelligent" features atop Linked Data so that it 
can be consumed will further block it's adoption.

 >
 > 3) All linked data ultimately boils down to literals, so our client
 > needs to understand how to render literals, which means, at the absolute
 > minimum, it can take a set of triples, either directly or from a
 > deconstructed JSON-LD document, and render them out to a simple table of
 > values.

Yes, a simple table or a form of graph. have you seen http://lodlive.it/ 
by the way?

 >
 > 4) We then "teach" our client to render common vocabulary entities in a
 > better way. Internally, this means overriding the default behaviour
 > whereby the client renders something instead of a general table of
 > values, and for additional data it can't account for, the
 > interface/template/whatever that is built to cater for that entity type,
 > tries to make allowances for additional data, such as rendering an
 > optional table at the bottom, offering a popup with the extra values, or
 > whatever is best, given the current context.
 >
 > 5) To make our client (paradoxically) both domain-specific and generic,
 > we accept that our API entry point is domain-specific by definition, so
 > we also teach our client how to render for our own needs. This means
 > that the method we use to teach our client rendering behaviours must
 > include functionality that overrides other behaviours defined earlier.
 > This may include overriding the rendering of specific data types, either
 > always, or only when certain conditions assert themselves. As a result
 > of this approach, our client starts with the baseline ability to render
 > any kind of linked data, but also becomes able to render a user
 > interface purpose-built for our application's domain-specific needs.

Simply displaying in a customized way may be reasonably simple. But what 
about teaching a client to give a great UX for operations. For example 
I've just recently implemented a table, where each cell contains a 
checkbox. When clicked a request is sent to the server. I have no idea 
how I would model that with any kind of hypermedia let alone creating a 
good UI dynamically based only on such model.

 >
 > 6) The world of linked data offers many, many vocabularies. It would be
 > a lot of work to try and teach our client to render every different
 > entity type in a user-friendly way. A possible solution is to come up
 > with a generic TBox vocabulary that gives us the tools to advertise a
 > suggested layout for a given entity type. Such a layout could include
 > positioning hints, sizing hints, etc. but would exclude anything to do
 > with theming, i.e. colours, fonts, etc. Furthermore, because a client
 > could override a given layout however it sees fit, it would mean we
 > could publish our data to the rest of the world and have them lay it out
 > in an acceptable way. We could still teach our client case-specific
 > overrides when we have custom needs. If there is already a vocabulary
 > that caters for my idea, let me know and I'll take a look.
 >
 > 7) Teaching a layout to the client could be done in either of two ways
 > (and these don't have to be mutually-exclusive.) First, you could simply
 > preconfigure the client to incorporate the layout vocabularies for, say,
 > schema.org <http://schema.org>, productontology.org
 > <http://productontology.org>, and your own domain-specific entities.
 > Second, it could be possible for linked data to advertise the location
 > of a layout vocabulary that supports it. Such metadata could easily be
 > ignored if not wanted.
 >
 > This solution would be applicable to Hydra as it could be used to help
 > us advertise to a client a way to lay out the Hydra functionality so
 > that it can be used easily by other people without them wondering why we
 > neglected the user experience. At the very least, it would be useful to
 > be able to publish hints, e.g. "These two actions are primary calls to
 > action." "These three are data maintenance options and should be
 > deprioritised." "This link is important and should be loaded
 > immediately." "This data should be available but not displayed." You get
 > the idea.
 >
 > On a tangential note, one thing I'm still not clear on is how I should
 > treat the advertisement of hypermedia controls with respect to a given
 > resource and the "purity" of that data, given that cacheability
 > requirements may conflict with the dynamic nature of the hypermedia
 > controls on offer and the context in which they're offered.
 >
 > Any comments, questions and/or criticisms are most welcome. Also, take a
 > look at a link Markus posted earlier regarding JSON-LD and web
 > components, as there are ideas there that may be relevant to a client
 > such as what I envisaged above.
 > 
http://updates.html5rocks.com/2015/03/creating-semantic-sites-with-web-components-and-jsonld

The above article could be a step towards a solution to your last 
paragraphs. How about instead of advertising layout, grouping, etc. a 
server would publish links to web components? Starting with a simplest 
example consider schema:Image, which optionally contains a thumbnail:

{
   "@type": "schema":ImageObject",
   "contentURL": "http://image/large",
   "thumbnail": {
     "contentURL": "http://image/small"
   }
}

A server would somehow advertise that it could be displayed with a 
custom web component <my-schema-image />, available at 
http://my.components.com/my-schema-image. The browser uses HTML import 
to download the component, generate the appropriate tag and pass the 
schema:ImageObject as attribute. The page could generate HTML like

<link rel="import" href="http://my.components.com/my-schema-image" />
<my-schema-image id="some-id" />
<script>
var image = { ... } // the image above
$('#some-id').image = image;
</script>

 From there the component takes over to render an the image. If present, 
the thumbnail would be displayed with a link to the full-size version (a 
modal maybe?):

<a href="http://image/large" target="_blank">
<img src="http://image/small">
</a>

 >
 > cheers,
 > Nathan Ridley
 >
 >
 > --
 > *Google*: axefrog@gmail.com <mailto:axefrog@gmail.com>
 > *Skype*: axefrog
 > *Twitter*: @NathanRidley
 > *Website*: http://axefrog.com / http://nathanridley.com
 > *Phone: *+61 (0) 475 072789

Received on Sunday, 15 March 2015 13:52:10 UTC