Re: Large Language Models, Visualizations, and Infographics

Civic Technology Community Group,

Exciting news! It appears that LIDA will be an open-source project: https://microsoft.github.io/lida/ .

Research and development are underway with respect to creating and styling data visualizations, e.g., charts and infographics, using dialogue systems and chatbots and I am starting to also think about enhancing conversational search for retrieving existing such items. With respect to civic technology and open government, end-users would be able to search for existing data visualizations pertinent to public-sector data, e.g., accounting, budgetary, and financial data, and, if their desired data visualizations did not already exist, they would be able to continue in dialogue to create and to style new ones for their needs.

Similarly, AI systems responding to users' questions (about public-sector data) would be able to search for existing multimedia resources before creating new ones as components of their multimodal responses.


Best regards,
Adam Sobieski

P.S.: I also recently updated a Wikianswers project proposal: https://meta.wikimedia.org/wiki/Wikianswers . A described vision for Wikimedia Commons, a multimedia content repository, also involves some of these same topics: utilizing dialogue systems and chatbots to search for, to create, and to style multimedia contents, e.g., 3D models, animations, audio, charts, diagrams, figures, graphs, images, infographics, maps, mathematics, photographs, tables, and video.

________________________________
From: Adam Sobieski <adamsobieski@hotmail.com>
Sent: Saturday, April 29, 2023 11:59 PM
To: public-civics@w3.org <public-civics@w3.org>
Subject: Large Language Models, Visualizations, and Infographics

Civic Technology Community Group,

Hello. Earlier, I indicated that "users will soon be able to ask natural-language questions and engage in multimodal dialogues about large-scale, public-sector financial, accounting, and budgetary data, receiving responses comprised of language, mathematics, charts, diagrams, figures, graphs, infographics, and tables."

Should these topics interest you, here are some recent scientific advancements pertaining to large language models, visualizations, and infographics:


LIDA: A Demo Video
Victor Dibia
https://vimeo.com/820968433


LIDA: Automatic Generation of Grammar Agnostic Visualizations and Infographics with Large Language Models (ChatGPT, GPT4)
Victor Dibia
https://newsletter.victordibia.com/p/lida-automatic-generation-of-grammar

"This post provides a high-level description of the design of a tool (LIDA) that supports users in automated data exploration and visualization/infographic generation using LLMs and image generation models (IGM’s).

"TLDR; LIDA provides the following capabilities.

  *   Data Summarization: Create a compact but information dense natural language representation of datasets, useful as grounding context for data operations with LLMs.
  *   Automatic Data Exploration: Given some raw data, come up with data exploration goals that make sense for this data. EDA for free!
  *   Grammar Agnostic Visualization Generation: Generate visualizations in any language, any visualization grammar (e.g., matplotlib, ggplot, altair etc).
  *   Infographic Generation: Generate stylized but “data-faithful” infographics, directly from data. Extensive applications in interactive data storytelling.
  *   Visualization Ops: Enables a set of operations on generated visualizations including - natural language based visualization refinement (.e.g change the x axis to .. translate chart to … zoom in by 50% etc), visualization explanation (code explanation, accessibility descriptions), visualization code self-evaluation (evaluation on dimensions such as aesthetics, compliance, type, transformation etc). Many applications here for accessibility, education and learning."


LIDA: A Tool for Automatic Generation of Grammar-agnostic Visualizations and Infographics using Large Language Models
Victor Dibia
https://arxiv.org/abs/2303.02927

"Systems that support users in the automatic creation of visualizations must address several subtasks - understand the semantics of data, enumerate relevant visualization goals and generate visualization specifications. In this work, we pose visualization generation as a multi-stage generation problem and argue that well-orchestrated pipelines based on large language models (LLMs) and image generation models (IGMs) are suitable to addressing these tasks. We present LIDA, a novel tool for generating grammar-agnostic visualizations and infographics. LIDA comprises of 4 modules - A SUMMARIZER that converts data into a rich but compact natural language summary, a GOAL EXPLORER that enumerates visualization goals given the data, a VISGENERATOR that generates, refines, executes and filters visualization code and an INFOGRAPHER module that yields data-faithful stylized graphics using IGMs. LIDA provides a python API, and a hybrid USER INTERFACE (direct manipulation and natural language) for interactive chart, infographics and data story generation."



Best regards,
Adam Sobieski