Re: shapes, forms, and footprints for building Linked Data apps

> On Jun 17, 2019, at 12:15 PM, Ruben Verborgh <ruben.verborgh@ugent.be> wrote:
> 
> Hi Irene,
> 
>> With respect to forms, SHACL already has some pre-built support for forms. For example, there is sh:order. There could have been more and the charter of the working group did cover this topic, but the working group ran out of time.
> 
> Thanks, this is very useful information!
> 
> And we didn’t make firm technology choices yet,
> so this can definitely influence our approach.
> 
>> Today, TopQuadrant uses SHACL and extensions to SHACL (the extension mechanism is part of the standard) to render forms in TopBraid EDG https://www.topquadrant.com/products/topbraid-enterprise-data-governance/. In doing this, we needed to address a few requirements that required some extensions.
> 
> Does that include specific UI controls for rendering certain fields?

Yes. There are “default" UI controls that correspond to the expected value of a field e.g., if it is a date, then, absent additional information, the control will be a date picker. The default can be overridden. 
> 
>> Not only different application may need to have different forms for the same underlying data, but different people using the same application may require different forms.
> 
> This one-to-many relationship between shapes and forms
> is indeed very crucial to us as well, for the same reasons.
> 
> However, this is what prompted us to separate shapes from forms;
> clearly delineating between shapes for machines,
> and forms that are UI for people.
> 
> I wonder how such a separation is realized with SHACL and extensions.
> For instance, the sh:order predicate you mention;
> to what parent object is it attached?
> 
> Are there examples we can compare?

Order is part of a property shape. A property shape is part of a node shape.

Btw, groups (sections) can also have an order.

One pattern we have used initially was to create a shapes graph (A) that has no UI information. Then, create another graph (B) that extends A by adding sh:order, etc., to the shapes in A. TopBraid EDG comes with many pre-build ontologies (expressed using SHACL) that define data assets (e.g., databases, tables, views, columns), technology assets (e.g., applications, servers, scripts), enterprise assets (e.g., organizations, processes, capabilities). Each of the ontologies has a graph of shapes and a graph of shapes-ux that add order, viewpoints, etc.

With this, you may have in the shapes graph something like:

edg:DatabaseTable
  Rdf:type owl:Class, sh:NodeShape ;
  edg:acronym "DB-TBL" ;
  edg:name "TABLE" ;
  rdfs:comment "A table of a database. This is a sub-type of 'Table'. Tables refer to their parent databases." ;
  rdfs:isDefinedBy <http://edg.topbraid.solutions/1.0/schema/datamodels> ;
  rdfs:label "Database Table" ;
  rdfs:subClassOf edg:ComplianceAspect ;
  rdfs:subClassOf edg:Processable ;
  rdfs:subClassOf edg:Table ;
  rdfs:subClassOf edg:Traceable ;
  sh:property edg:DatabaseTable-columnOf-inverse ;
  sh:property edg:DatabaseTable-foreignKey ;
  sh:property edg:DatabaseTable-tableOf ;
.

edg:DatabaseTable-tableOf
  rdf:type sh:PropertyShape ;
  sh:path edg:tableOf ;
  rdfs:isDefinedBy <http://edg.topbraid.solutions/1.0/schema/datamodels> ;
  sh:class edg:Database ;
  sh:maxCount 1 ;
.

And then in the ux-shapes

edg:DatabaseTable-tableOf
  sh:group edg:DataStructurePropertiesGroup ;
  sh:order "50"^^xsd:decimal ;
.

This works perfectly well. However, we found that, for our purposes, such separation was an unnecessary overhead in the ongoing development and maintenance of these ontologies. It was not a large overhead, but it was unnecessary. This made us decide that we did not want to enforce it. In the context of some other needs, it may be more useful.

Organizations that use EDG, customize the models is based on. Customization involves modifying and creating new shapes. Customization process is as follows:

If they do not want to use a property because this is not a piece of data they will ever track, they deactivate a property shape. SHACL has sh:deactivated.
If they want a property but want it in another group or with another order within that group or with some other differences that pertain only to the UI, they would still need to deactivate the shape and then create a new one for the same property. 
The layering of deactivation is necessary because we ship new releases that contain the ontologies and want any changes user will make to be forward compatible. Their changes are in their own graph and the statement that some shape was deactivated will survive the upgrade.
If they want to add a property, they create a new shape

The above means that in a process of configuration, users would have to re-enter the value constraints only because they want to move a property to another section. Some other design may not require this. However, practically speaking, in our experience, this never happens for such “general shapes”. It may happen in the context of a role specific display which is explained below.

Moving on to roles, there may be a shape describing how database tables are to be presented to a certain group of people, let’s say:

example:DatabaseTableSMEView a sh:NodeShape;
dash:applicableToClass edg:DatabaseTable;
dash:defaultViewForRole edg:subjectMatterExpert

Dash: namespace has extensions to SHACL.

Typically, to create such shape a user would use a wizard that would let them pick among all properties for which a property shape exists for edg:DatabaseTable. The dialog below shows a few, but there are many more if I were to scroll down.


User would normally only pick those that should be presented in a view they are designing. These shapes are then copied over to the view shape as blank nodes, so that any adjustment could be made “locally” e.g.,

example:DatabaseTableSMEView a sh:NodeShape;
dash:applicableToClass edg:DatabaseTable;
dash:defaultViewForRole edg:subjectMatterExpert;
sh:property [
      rdf:type sh:PropertyShape ;
      sh:path edg:tableOf ;
      sh:class edg:Database ;
      sh:group edg:DataStructurePropertiesGroup ;
      sh:maxCount 1 ;
      sh:order "50"^^xsd:decimal ;
    ] ;
.

It does not have to be this way. Such role-specific shape could also be

example:DatabaseTableSMEView a sh:NodeShape;
dash:applicableToClass edg:DatabaseTable;
dash:defaultViewForRole edg:subjectMatterExpert;
edg:DatabaseTable-tableOf.

But the wizard creates blank nodes (“local” property shapes) to make it easier to customize locally.


> 
>> To that end, we have extended SHACL to support role-specific views. The extensions is quite simple - a NodeShape can have an additional property with a value corresponding to user’s role. Then, this shape is selected (among all applicable shapes) for presenting information to a user in that role. 
> 
> So, a form is considered a specialization of a shape?

Not explicitly, there is no class ;Form if this is what you mean, but essentially I think, yes.
> 
>> Other extensions address the fact that a view presentation of data may need to be different from the edit presentation, provide  instructions for embedding and creating “related” resources (e.g., a person may have multiple addresses with people and addresses being separate related resource, from a form perspective, a user will want to see and create addresses as part of a person’s form), etc.
> 
> These might probably go into what we call “footprints”.

Probably. 

Typically, users want these to be shown as a table because they want to see more data than is possible in a list. For example, in the screenshot below attributes are shown and can be created as part of working with the information about a logical entity. It shows a member of a edg:LogicalEntity class called “Movie Rental” with the resources representing its attributes.



In your example, this could be a list of people that belong to a group where columns show their e-mail addresses, phone numbers, etc. They do not have to be completely dependent resources, but they are dependent in the context of this view.

In the context of EDG, our goal is model-driven generation of views. In the context of other applications that are using information in EDG, the choice is left open. 

EDG provides GraphQL interface with introspection. It means people who formulate GraphQL queries could fetch any information exposed through the model - at any level of nesting with any order of properties. It is a matter of how much modeling they want to do. Many developers today do not understand or want to do modeling and prefer flexibility of describing the structures they want for their applications in GraphQL.

> 
>> Since UI applications today use JSON, we are using SHACL to dynamically generate enhanced GraphQL Schemas. This is described in detail at https://www.topquadrant.com/technology/graphql/.
> 
> Super, this is of great interest to me as well,
> and there seems to be some correspondence to
> https://ruben.verborgh.org/blog/2019/06/17/shaping-linked-data-apps/#shapes-APIs
> 
> 
> 
> In general, it seems that TopQuadrant has chosen to stay close to shapes,
> whereas we thought that we would restrict the scope of shapes to just structure,
> and then deliberately using separate concepts for forms and footprints.
> The idea being that they are different things edited by different people.
> Will be very interesting to compare these angles.

Sure, this sounds interesting. We have been working on the approaches to address these requirements for well over 10 years now - going back and forth and trying different designs to see what can practically satisfy different requirements. This is where we ended for now. Holger could say more.

> 
> Thanks,
> 
> Ruben