Re: Large Language Models, Visualizations, and Infographics

Thanks for the update Adam,

Seems like an interesting project. Where was it announced that it was going
to be open sourced? I wonder what license they plan to use, or for that
matter when they plan to release the code.  Curious what it would need to
run (aside from a device running python).

It definitely seems like it could be a pretty powerful tool. Was
interesting watching that video.

Would be fascinating to see the accessibility work that they show get
factored into a dynamic, accessible SVG file. Could very much solve a
problem about how to provide accessible, dynamic data visualizations.

Still, how do we avoid the "lies, damn, lies & statistics”. Given that most
folks accessing this will not have a background in statistics.  Will this
just propagate more compelling stories (that aren’t based on fact).

So how would this potentially plug into an open data set like DKAN or CKAN?
How would you see the data there being exposed to users?

Mike


Mike Gifford, Senior Strategist, CivicActions
Drupal Core Accessibility Maintainer
https://civicactions.com    |  https://accessibility.civicactions.com
http://twitter.com/mgifford |  http://linkedin.com/in/mgifford

On May 11, 2023 at 9:33:23 PM, Adam Sobieski (adamsobieski@hotmail.com)
wrote:

Civic Technology Community Group,

Exciting news! It appears that LIDA will be an open-source project:
https://microsoft.github.io/lida/ .

Research and development are underway with respect to creating and styling
data visualizations, e.g., charts and infographics, using dialogue systems
and chatbots and I am starting to also think about enhancing conversational
search for retrieving existing such items. With respect to civic technology
and open government, end-users would be able to search for existing data
visualizations pertinent to public-sector data, e.g., accounting,
budgetary, and financial data, and, if their desired data visualizations
did not already exist, they would be able to continue in dialogue to create
and to style new ones for their needs.

Similarly, AI systems responding to users' questions (about public-sector
data) would be able to search for existing multimedia resources before
creating new ones as components of their multimodal responses.


Best regards,
Adam Sobieski

P.S.: I also recently updated a *Wikianswers* project proposal:
https://meta.wikimedia.org/wiki/Wikianswers . A described vision for
Wikimedia Commons, a multimedia content repository, also involves some of
these same topics: utilizing dialogue systems and chatbots to search for,
to create, and to style multimedia contents, e.g., 3D models, animations,
audio, charts, diagrams, figures, graphs, images, infographics, maps,
mathematics, photographs, tables, and video.

------------------------------
*From:* Adam Sobieski <adamsobieski@hotmail.com>
*Sent:* Saturday, April 29, 2023 11:59 PM
*To:* public-civics@w3.org <public-civics@w3.org>
*Subject:* Large Language Models, Visualizations, and Infographics

Civic Technology Community Group,

Hello. Earlier, I indicated that "users will soon be able to ask
natural-language questions and engage in multimodal dialogues about
large-scale, public-sector financial, accounting, and budgetary data,
receiving responses comprised of language, mathematics, charts, diagrams,
figures, graphs, infographics, and tables."

Should these topics interest you, here are some recent scientific
advancements pertaining to large language models, visualizations, and
infographics:


*LIDA: A Demo Video*
Victor Dibia
https://vimeo.com/820968433


*LIDA: Automatic Generation of Grammar Agnostic Visualizations and
Infographics with Large Language Models (ChatGPT, GPT4)*
Victor Dibia
https://newsletter.victordibia.com/p/lida-automatic-generation-of-grammar

"This post provides a high-level description of the design of a tool (LIDA)
that supports users in automated data exploration and
visualization/infographic generation using LLMs and image generation models
(IGM’s).

"TLDR; LIDA provides the following capabilities.

   - Data Summarization: Create a compact but information dense natural
   language representation of datasets, useful as grounding context for data
   operations with LLMs.
   - Automatic Data Exploration: Given some raw data, come up with data
   exploration goals that make sense for this data. EDA for free!
   - Grammar Agnostic Visualization Generation: Generate visualizations in
   any language, any visualization grammar (e.g., matplotlib, ggplot, altair
   etc).
   - Infographic Generation: Generate stylized but “data-faithful”
   infographics, directly from data. Extensive applications in interactive
   data storytelling.
   - Visualization Ops: Enables a set of operations on generated
   visualizations including - natural language based visualization refinement
   (.e.g change the x axis to .. translate chart to … zoom in by 50% etc),
   visualization explanation (code explanation, accessibility descriptions),
   visualization code self-evaluation (evaluation on dimensions such as
   aesthetics, compliance, type, transformation etc). Many applications here
   for accessibility, education and learning."



*LIDA: A Tool for Automatic Generation of Grammar-agnostic Visualizations
and Infographics using Large Language Models*
Victor Dibia
https://arxiv.org/abs/2303.02927

"Systems that support users in the automatic creation of visualizations
must address several subtasks - understand the semantics of data, enumerate
relevant visualization goals and generate visualization specifications. In
this work, we pose visualization generation as a multi-stage generation
problem and argue that well-orchestrated pipelines based on large language
models (LLMs) and image generation models (IGMs) are suitable to addressing
these tasks. We present LIDA, a novel tool for generating grammar-agnostic
visualizations and infographics. LIDA comprises of 4 modules - A SUMMARIZER
that converts data into a rich but compact natural language summary, a GOAL
EXPLORER that enumerates visualization goals given the data, a VISGENERATOR
that generates, refines, executes and filters visualization code and an
INFOGRAPHER module that yields data-faithful stylized graphics using IGMs.
LIDA provides a python API, and a *hybrid* USER INTERFACE (direct
manipulation and natural language) for interactive chart, infographics and
data story generation."



Best regards,
Adam Sobieski