Re: [w3ctag/design-reviews] Incubation: WebMCP (Issue #1238)

domfarolino left a comment (w3ctag/design-reviews#1238)

Yesterday we discussed WebMCP at the TAG Atlantic breakout meeting—thanks to all of the TAG members that participated, and took the time to review the proposal so far!

One thing that keeps coming up over and over is: why is our solution shaped like an explicit capability layer / agent pathway to site code, as opposed to just "more HTML semantics" to help agents do everything through the UI? Some members of the TAG are happy with our approach, but not basing the proposal entirely on more UI semantics has been the primary source of TAG pushback we've received. (I'll add the caveat that a formal position has not yet been reached, and the TAG is not explicitly demanding that a "more semantic HTML" approach be pursued).

With that, I'd like to gently ask: if the TAG proposes a negative resolution based substantially on our API's lack of "more HTML semantics", that the TAG please provide evidence as to why this is believed to be a sufficient alternative. A sufficient alternative not to just what we cooked up with WebMCP, but to what many corners of the industry have independently converged on as a battle-tested approach to agents interacting with UI applications.

To add more color:
 - Frontier AI labs like OpenAI, Anthropic, and Google have landed on a similar protocol (MCP) for using programmatic interfaces
 - Those same stakeholders struggle even more to actuate UI, and have shown interest in WebMCP. In a related use case, we've developed a [similar tool-calling approach](https://apps.extensions.modelcontextprotocol.io/api/types/app.AppToolResult.html) inside of MCP _Apps_ to perform actions inside web UI.
 - Open-source extensions like WebBrain [have attested](https://www.webbrain.one/blog/webmcp-integration) to high cost, latency, and inaccuracy when actuating content through the UI. They integrated with WebMCP as soon as they could. We get similar feedback from other developers.
 - Apple's Siri AI integration actuates native macOS apps exclusively through AppIntents, which [basically mirrors imperative WebMCP](https://developer.apple.com/documentation/appintents/acceleratingappinteractionswithappintents). This is despite macOS apps being notoriously accessible, and UI frameworks like SwiftUI being inherently semantic and declarative. This allows for background operations to be performed, which we could support on the web with Service Workers.
 - Microsoft (to support Copilot in Edge) and Google (to support Gemini in Chrome) found that WebMCP leads to far better user experiences over AI actuation. Mozilla has agreed with us that an explicit capability layer beyond HTML semantics seems required, and are supportive _specifically_ of the imperative API (which they have [opinions](https://github.com/webmachinelearning/webmcp/issues/236) on).
 - Independent developers have run many studies showing that WebMCP leads to more accurate user actions and journeys, while being "3-8x cheaper" (tokens) and "4-5x faster" than UI actuation. I'll link to this report now, but there are more coming: https://claude.ai/code/artifact/ff487f98-ef0b-45bc-a0f9-4574e1c157a3

I'll be the first to say that we can do a better job documenting why we believe the gaps of agent UI actuation are "capability layer" shaped, not just "need more semantic UI" shaped. And we're working on that. But if the TAG does feel strongly about a semantic UI route, I'd kindly ask that some evidence be supplied that the current industry SOTA on which all stakeholders have converged is either overkill, or can equally be matched by more HTML attributes and kin. And why the drawbacks of such an approach—no service worker integration, and possibly not supporting in-page agents in iframes—are worth it.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/w3ctag/design-reviews/issues/1238#issuecomment-5207638057
You are receiving this because you are subscribed to this thread.

Message ID: <w3ctag/design-reviews/issues/1238/5207638057@github.com>