RE: Explanation, Mechanistic Reasoning, and Abstraction: Hypertext and Hypermodels

Dave Raggett,
David Booth,
Nicolas Chauvat,
All,

In my opinion, some interesting analogues can be found in the computer vision and perception domains where, for some tasks, adaptive and task-specific levels of detail are desired for outputs.

With respect to visual scene understanding, for a scene (imagery, video, or 3D), a system can recognize features, objects, events, and activities. But which features, objects, events, and activities are instantaneously most relevant? Which level of detail over the contents of the entire scene is most appropriate for a task or purpose?

In these regards, one can explore Scene Graph Representation and Learning (https://cs.stanford.edu/people/ranjaykrishna/sgrl/index.html) to see how semantic graphs can be provided for the contents of visual scenes. But which graphs from a set of possible graphs are instantaneously most relevant? Which level of detail is most appropriate for a task or purpose?

From these analogues from the computer vision and perception domains, we can see a bridge from situations where one best or correct output for some input is desired to situations where attention and task context play a role in determining which outputs (or working space contents) are best or correct. Which output is best for a particular task or purpose? Which understood scene and recognized contents (features, objects, events, and activities) are at the best level of detail? Which learned (causal) model is at the best level of detail? How can a deep learning architecture move through a trajectory of scene understandings or of (causal) models, applying transformations or category-theoretical functions to these contents?

Perhaps deep learning architectures will soon be able to use attention and/or task context to transform or apply category-theoretical functions to contents in a working space, be these contents recognized from visual scenes or the representations of (causal) models. Perhaps deep learning architectures will soon be able to zoom in to and out of parts of visual scenes or of (causal) models, providing users with adaptive and task-specific levels of detail.

Perhaps question-answering systems can elucidate these processes on (causal) model contents in a working space; each transformation or category-theoretical function on a (causal) model could be described as providing an answer to an increasingly detailed question about the explanation or (causal) model.

Perhaps, as bidirectional mappings are being explored between natural language and visual contents (imagery, video, and 3D), in the near future, we might see deep learning architectures which can provide mappings between natural language explanations and (causal) models.

In addition to how to conveniently navigate between models, there is the matter of how deep learning architectures might conveniently do so.


Best regards,
Adam Sobieski

From: David Booth<mailto:david@dbooth.org>
Sent: Saturday, November 27, 2021 1:05 PM
To: semantic-web@w3.org<mailto:semantic-web@w3.org>
Subject: Re: Explanation, Mechanistic Reasoning, and Abstraction: Hypertext and Hypermodels

On 11/27/21 7:00 AM, Adam Sobieski wrote:
. . .
>  1. The robot caused the elevator to arrive.
>  2. The robot pressed the button which caused the elevator to arrive.
>  3. The robot used its arm to press the button which caused the elevator
>     to arrive.
. . .
> Any thoughts on these topics?

Yes.  All of the above examples are reasonable ways to model what
happened, and each model would be appropriate for a particular purpose.

I think we (still) need convenient ways both to express the
relationships between those models and to conveniently shift from one
model to another when using them.  This is analogous to using
(aggregate) objects in programming.  For one purpose we might manipulate
an object as a whole, but for another purpose we might need to open up
that object to see and manipulate its constituent parts -- a greater
level of detail.  Or vice versa: we may have the constituent parts, and
wish to "bless" those parts to become a coherent object that can be
manipulated as a whole.

I view ontologies and rules as the basic building blocks for addressing
this problem: ontologies can define declarative relationships between
models, and rules can define operational conversion between models.
However, in spite of the fact that we've had ontologies and rules for
many years, I think this field is still in its infancy in terms of
having strong patterns and convenient languages for using them with RDF
to easily navigate between modeling levels.  "Levels" is not quite the
right word though, because different models form a network, not a single
hierarchy.

I personally think N3 shows the greatest promise so far, toward making
rules convenient to use with RDF, and I'm grateful that there's a core
group of dedicated folks who have been working diligently to develop,
implement, and standardize it:
https://www.w3.org/community/n3-dev/

It would be wonderful if some bright young minds could take a fresh look
at the problem of how to conveniently navigate between models, and come
up with some fresh bold new ideas.   Or perhaps even some stale modest
old ideas that would still make incremental progress.  :)

David Booth