Re: Web Charts

Civic Technology Community Group,

In the postscript of the previous email, I hyperlinked to a WHATWG GitHub issue about Web Charts (https://github.com/whatwg/html/issues/9295) which has since unfolded so as to include discussion of some innovative accessibility ideas.

Text templates, e.g., like those used in software like Handlebars<https://handlebarsjs.com/>, could be of use for generating dynamic descriptions of charts. Charts could generate detailed natural-language descriptions of themselves. These descriptions could be processed by AI systems. AI systems would, then, be able to answer questions about and to engage in dialogues about the charts.

AI systems can, today, produce and provide charts in their multimodal responses to end-users' natural-language questions and search queries. Soon, end-users will also be able to search for, create, and style charts via natural-language interactions with AI systems. Soon, end-users will also be able to ask questions about and engage in dialogues about the contents of charts in documents, including in accessibility contexts.


Best regards,
Adam

________________________________
From: Adam Sobieski <adamsobieski@hotmail.com>
Sent: Monday, May 15, 2023 5:22 AM
To: public-civics@w3.org <public-civics@w3.org>
Subject: Web Charts

Civic Technology Community Group,

We have been recently discussing that developments in AI, such as LLMs and the LIDA project, can be utilized for answering those natural-language questions that accountants, auditors, analysts, bureaucrats, comptrollers, public officials, legislators, oversight committees, and members of their staffs, as well as the public, journalists, and government watchdog organizations, might have about collections of documents or data. Those kinds of public-sector data in scope include, but are not limited to, accounting, budgetary, and financial data.

During the course of our discussion, some scenarios of concern were indicated pertaining to statistical charts. Instead of having to represent data visualizations as images or SVG, I indicated that new markup would be useful, making possible new algorithms with which to detect anomalous, misleading, or incorrect usages of data visualizations for both AI systems and end-users. With representations of charts which included their data and/or queries, ML or AI algorithms would better be able to detect anomalous, misleading, or incorrect usages of visualizations as occurring in natural-language hypertext document contexts.

Here are some preliminary ideas with respect to the envisioned new markup, Web Charts, expressed in the form of a sequence of examples.


Example 1: This example shows a basic chart comprised of inline data and a script which processes that data into document structure.

<chart>
  <data type="text/csv">...</data>
  <script src="https://cdn.webcache.org/piechart.js" />
</chart>

Example 2: This example shows a basic chart comprised of inline data, a script which processes that data into document structure, and custom styling.

<chart>
  <data type="text/csv">...</data>
  <script src="https://cdn.webcache.org/piechart.js" />
  <style src="custom.css" />
</chart>

Example 3: This example shows a chart which, instead of providing inline data, provides a SPARQL query and service from which to retrieve data.

<chart>
  <query service="https://semanticweb.org" method="post" accept="text/csv" type="application/sparql-query">...</query>
  <script src="https://cdn.webcache.org/piechart.js" />
</chart>

Example 4: This example shows a means of providing extensible chart semantics or metadata.

<chart>
  <metadata rel="about" type="application/ld+json">...</metadata>
  <data type="text/csv">...</data>
  <script src="https://cdn.webcache.org/piechart.js" />
</chart>

Example 5: These examples show possibilities for adding a text description to a chart.

<chart alt="This is a text description of a pie chart.">
  <data type="text/csv">...</data>
  <script src="https://cdn.webcache.org/piechart.js" />
</chart>

<chart>
  <metadata rel="alt" type="text/plain" lang="en">This is a text description of a pie chart.</metadata>
  <data type="text/csv">...</data>
  <script src="https://cdn.webcache.org/piechart.js" />
</chart>

Example 6: This example provides alternative versions of a chart, in this case multiple PNGs, in a fallback mechanism.

<chart>
  <fallback>
    <source media="..." type="image/png" src="piechart-1a.png" />
    <source media="..." type="image/png" src="piechart-1b.png" />
  </fallback>
  <query service="https://semanticweb.org" method="post" accept="text/csv" type="application/sparql-query">...</query>
  <script src="https://cdn.webcache.org/piechart.js" />
</chart>

Example 7: These examples show that markup for reusable charts could also be stored in separate resources from those documents including the charts.

<chart src="piechart1.webc" />

<img src="piechart1.webc" />


Notes:

  1.  As envisioned, Web Charts would be interoperable with every other Web technology. Web Charts could realize data-driven charts utilizing HTML, SVG, Canvas2D, WebGL, etc.

Questions:

  1.  What about specifying caching, updating, or refreshing behavior with respect to query response data for a chart? Attributes on the <chart> element and/or protocol response headers?
  2.  What about visualizing streaming data, e.g., stock market data, with Web Charts?
  3.  What about including provenance-related metadata, e.g., a query and endpoint, while simultaneously providing inline data to be utilized in producing the chart?
  4.  What about configuration or options for a script, e.g., multiple types of pie charts (a doughnut variation)? Would these topics be matters for style or would a new <options> element, e.g., for JSON data, be useful?
  5.  What about specifying alternatives for a SPARQL service URL?
  6.  What about scenarios involving the clipboarding of or dragging-and-dropping of charts, e.g., between browser tabs or between browsers and document-authoring software?


Best regards,
Adam Sobieski

P.S.: https://github.com/whatwg/html/issues/9295