Re: LLMs and Agents usage in the W3C CCG

This is a rich and important conversation around the position W3C should assume regarding LLMs and their agents involved or participating in standards development.  It’s apparent that this cannot be ignored. To wait and see where things go isn’t a viable option. To that extent I share the perspective that W3C needs to take a leadership role in the approach it chooses to adopt. The extent to which the organization has the resources and bandwidth to put toward this will necessarily be one vector that shapes the decisions made.

I’m appreciative of the argument for LLMs enabling broader participation from non-native English speakers in the deliberations and contributions to spec development work. There are talented people the world over who likely are limited in their ability to participate and for whom AI translation and transcription tools would be very helpful. I’m less inclined to consider usage of LLMs in this context requires a disclosure statement.  More on ways to approach that and related disclosure usage at the close of this posting.

Amir has quickly gotten to some core issues underlying the discussion - how do we ensure the maintenance, accountability, provenance, and trust in the contributions to the collective work of W3C when it’s done largely by AI agents or LLMs with a few prompts?  Similarly, the infrastructure required to implement LLM or AI agents to do this work and to measure and assess how well it is being done is non-trivial. We spent considerable effort to maintain open-source video conference services and implement some form of rudimentary transcription services. The magnitude and costs to pursue this led to a thoughtful, well documented and data driven assessment of commercial alternatives, given the time and cost to continue to in-house effort which was likely to be expensive. (in time and money).

I think we have an obligation to be transparent and disclose LLM use when it's beyond simply facilitating more accurate, grammatical communication. I've used it, like many of you as a 'thought partner', and found it helpful. It's notable when doing so in an area I actually do know something about and observe the LLM has made some serious mistakes or unwarranted assumptions. The concern is someone who is in fact a true newbie won't see it, wasting their time and others.

I'm currently an advocate of requiring, for now at least, no LLM usage beyond grammatical fixes and vocabulary, to things posted to this discussion list. As they tools get better and we get more comfortable a pivot to a different stance might in order. But for now I'm in the no LLM written posts to the listserv camp.

I do have a suggestion on how disclosure might be best approached to achieve the need for  transparency and accountability during this transition period.

The idea stems from the work of Creative Commons in documenting the licensing preferences of the author of creative works. They spent time coming up with some elegant simple declaratives and associated visual 'stamps' that were backed up by a link to a fuller declarative statement to specify the intended meaning.  Most of you I think are familiar with them.

CC License Icons/stamps:

  *   Attribution (BY): A person icon (user silhouette) indicating credit must be given to the creator.
  *
NonCommercial (NC): A "$$" symbol with a diagonal line through it, indicating the work cannot be used for commercial purposes.
  *
NoDerivatives (ND): An equals sign ("==") symbol, indicating the work must be used in its original form without modifications.
  *   ShareAlike (SA): A circular arrow icon (refresh symbol), indicating that adaptations must be shared under the same license.
  *   CC Zero (CC0) / Public Domain: A "CC" logo combined with a "0" or a stylized "No Rights Reserved" icon, indicating the creator has dedicated the work to the public domain

and, Common License Combinations

CC BY: CC + Person

  *   CC BY-SA: CC + Person + Circular Arrow
  *
CC BY-NC: CC + Person + $$ Sign
  *   CC BY-ND: CC + Person + Equals Sign
  *
CC BY-NC-SA: CC + Person +$ Sign + Circular Arrow
  *
CC BY-NC-ND: CC + Person + $ Sign + Equals Sign

The associated images that link to the explanatory pages can be found at the CC, and an example of the CC equivalent of the MIT Open Source software license, CC BY is here<https://creativecommons.org/licenses/by/4.0/>

I do not mean to suggest this is easy or can be done quickly. The elegance of the CC licensing representation hides several years of hard work by a dedicated team of developers, lawyers, and others. My initial suggestion is formation of a workgroup to explore what W3C should do regarding disclosure and transparency when using output significantly developed in collaboration with an LLM or associated AI agents.  An initial perhaps stop gap work product might be a stamp/simple descriptor of the degree to which LLM use contributed to the outcome directly, and not just as a preliminary step in the thought partnership with a human who entirely writes what they learned both from the LLM and from their own experiences.

We have an opportunity here that not only might guide our future work, but others struggling with these questions in theirs.

Cheers,
  Phil

Senior Consultant - VC/LERs & Digital Identities
Enterprise Technologies<https://tech.asu.edu/>, Engineering - Cloud Systems
Arizona State University
e: pdlong2@asu.edu<mailto:pdlong2@asu.edu>
bio: https://about.me/phillip.long<https://about.me/phillip.long>
--
LER Network Consultant
<https://www.uschamberfoundation.org/solutions/workforce-development-and-training/t3-innovation-network>T3 Innovation Network
<https://www.uschamberfoundation.org/solutions/workforce-development-and-training/t3-innovation-network>US Chamber of Commerce Foundation<https://www.uschamberfoundation.org/solutions/workforce-development-and-training/t3-innovation-network>
e: phil@rhzconsulting.com<mailto:phil@rhzconsulting.com>