Re: Large Language Models, Visualizations, and Infographics

Mike,

On the https://microsoft.github.io/lida/ page, in the FAQs section, it was announced that "LIDA is an ongoing project and there is work underway to package and release the code. Stay tuned!" This text is hidden behind an expanding accordion view widget with the heading: "Source Code?".

To your next question, detecting anomalous, misleading, or incorrect usages of visualizations, brainstorming, we might consider providing data and metadata for visualizations, e.g., charts and infographics, to enhance the capability for AI/ML algorithms to better analyze them, as they occur in document contexts. That is, we can consider and explore dynamic and accessible models and markups for representing visualizations instead of considering them to have to be images or pictures.

It also seems to be advantageous to consolidate visualizations, for users and AI systems to search for existing resources before creating new ones. With consolidated visualizations, algorithms would, I think, be able to more efficiently detect events resembling, for example, that, in 99% of documents, a specific visualization was used in one way, and, in 1%, or in one document, in another way.

Do any other participants here have any thoughts about detecting anomalous, misleading, or incorrect uses of visualizations as they occur in document contexts? These algorithms could be useful for content authoring and design tools and also for evaluating AI systems and their multimodal responses. These algorithms might also be able to access and process those underlying data, and/or queries used to obtain those data, of visualizations as well.

With respect to CKAN (https://ckan.org/,  https://ckan.org/government), DKAN (https://getdkan.org/), and technical interoperability topics, I will explore these matters in greater detail, upcoming, and then respond here.


Best regards,
Adam

________________________________
From: Mike Gifford <mike.gifford@civicactions.com>
Sent: Thursday, May 11, 2023 10:18 PM
To: public-civics@w3.org <public-civics@w3.org>; Adam Sobieski <adamsobieski@hotmail.com>
Subject: Re: Large Language Models, Visualizations, and Infographics

Thanks for the update Adam,

Seems like an interesting project. Where was it announced that it was going to be open sourced? I wonder what license they plan to use, or for that matter when they plan to release the code.  Curious what it would need to run (aside from a device running python).

It definitely seems like it could be a pretty powerful tool. Was interesting watching that video.

Would be fascinating to see the accessibility work that they show get factored into a dynamic, accessible SVG file. Could very much solve a problem about how to provide accessible, dynamic data visualizations.

Still, how do we avoid the "lies, damn, lies & statistics”. Given that most folks accessing this will not have a background in statistics.  Will this just propagate more compelling stories (that aren’t based on fact).

So how would this potentially plug into an open data set like DKAN or CKAN? How would you see the data there being exposed to users?

Mike


Mike Gifford, Senior Strategist, CivicActions
Drupal Core Accessibility Maintainer
https://civicactions.com<https://civicactions.com/>    |  https://accessibility.civicactions.com
http://twitter.com/mgifford |  http://linkedin.com/in/mgifford


On May 11, 2023 at 9:33:23 PM, Adam Sobieski (adamsobieski@hotmail.com<mailto:adamsobieski@hotmail.com>) wrote:

Civic Technology Community Group,

Exciting news! It appears that LIDA will be an open-source project: https://microsoft.github.io/lida/ .

Research and development are underway with respect to creating and styling data visualizations, e.g., charts and infographics, using dialogue systems and chatbots and I am starting to also think about enhancing conversational search for retrieving existing such items. With respect to civic technology and open government, end-users would be able to search for existing data visualizations pertinent to public-sector data, e.g., accounting, budgetary, and financial data, and, if their desired data visualizations did not already exist, they would be able to continue in dialogue to create and to style new ones for their needs.

Similarly, AI systems responding to users' questions (about public-sector data) would be able to search for existing multimedia resources before creating new ones as components of their multimodal responses.


Best regards,
Adam Sobieski

P.S.: I also recently updated a Wikianswers project proposal: https://meta.wikimedia.org/wiki/Wikianswers . A described vision for Wikimedia Commons, a multimedia content repository, also involves some of these same topics: utilizing dialogue systems and chatbots to search for, to create, and to style multimedia contents, e.g., 3D models, animations, audio, charts, diagrams, figures, graphs, images, infographics, maps, mathematics, photographs, tables, and video.

________________________________
From: Adam Sobieski <adamsobieski@hotmail.com<mailto:adamsobieski@hotmail.com>>
Sent: Saturday, April 29, 2023 11:59 PM
To: public-civics@w3.org<mailto:public-civics@w3.org> <public-civics@w3.org<mailto:public-civics@w3.org>>
Subject: Large Language Models, Visualizations, and Infographics

Civic Technology Community Group,

Hello. Earlier, I indicated that "users will soon be able to ask natural-language questions and engage in multimodal dialogues about large-scale, public-sector financial, accounting, and budgetary data, receiving responses comprised of language, mathematics, charts, diagrams, figures, graphs, infographics, and tables."

Should these topics interest you, here are some recent scientific advancements pertaining to large language models, visualizations, and infographics:


LIDA: A Demo Video
Victor Dibia
https://vimeo.com/820968433


LIDA: Automatic Generation of Grammar Agnostic Visualizations and Infographics with Large Language Models (ChatGPT, GPT4)
Victor Dibia
https://newsletter.victordibia.com/p/lida-automatic-generation-of-grammar

"This post provides a high-level description of the design of a tool (LIDA) that supports users in automated data exploration and visualization/infographic generation using LLMs and image generation models (IGM’s).

"TLDR; LIDA provides the following capabilities.

  *   Data Summarization: Create a compact but information dense natural language representation of datasets, useful as grounding context for data operations with LLMs.
  *   Automatic Data Exploration: Given some raw data, come up with data exploration goals that make sense for this data. EDA for free!
  *   Grammar Agnostic Visualization Generation: Generate visualizations in any language, any visualization grammar (e.g., matplotlib, ggplot, altair etc).
  *   Infographic Generation: Generate stylized but “data-faithful” infographics, directly from data. Extensive applications in interactive data storytelling.
  *   Visualization Ops: Enables a set of operations on generated visualizations including - natural language based visualization refinement (.e.g change the x axis to .. translate chart to … zoom in by 50% etc), visualization explanation (code explanation, accessibility descriptions), visualization code self-evaluation (evaluation on dimensions such as aesthetics, compliance, type, transformation etc). Many applications here for accessibility, education and learning."


LIDA: A Tool for Automatic Generation of Grammar-agnostic Visualizations and Infographics using Large Language Models
Victor Dibia
https://arxiv.org/abs/2303.02927

"Systems that support users in the automatic creation of visualizations must address several subtasks - understand the semantics of data, enumerate relevant visualization goals and generate visualization specifications. In this work, we pose visualization generation as a multi-stage generation problem and argue that well-orchestrated pipelines based on large language models (LLMs) and image generation models (IGMs) are suitable to addressing these tasks. We present LIDA, a novel tool for generating grammar-agnostic visualizations and infographics. LIDA comprises of 4 modules - A SUMMARIZER that converts data into a rich but compact natural language summary, a GOAL EXPLORER that enumerates visualization goals given the data, a VISGENERATOR that generates, refines, executes and filters visualization code and an INFOGRAPHER module that yields data-faithful stylized graphics using IGMs. LIDA provides a python API, and a hybrid USER INTERFACE (direct manipulation and natural language) for interactive chart, infographics and data story generation."



Best regards,
Adam Sobieski