Minutes from W3C M&E IG call: Report on W3C Workshop on Web Games

Dear all,

The minutes from the Media & Entertainment Interest Group call on Tuesday 3rd September are now available [1], and copied below. We discussed outcomes from the W3C Workshop on Web Games, which was held in June [2].

Many thanks to Francois for presenting, and all who joined the discussion.

A recording of the presentation and discussion is also available [3].

Kind regards,

Chris (Co-chair, W3C Media & Entertainment Interest Group)

[1] https://www.w3.org/2019/09/03-me-minutes.html
[2] https://www.w3.org/2018/12/games-workshop/report.html
[3] https://media.w3.org/2019/09/meig-2019-09-03.mp4

--

W3C
- DRAFT -
Media and Entertainment IG
03 Sep 2019

Agenda

    https://lists.w3.org/Archives/Public/public-web-and-tv/2019Aug/0004.html

Attendees

Present
    Francois_Daoust, Chris_Needham, Kaz_Ashimura, Takahiro_Kumekawa, Barbara_Hochgesang, Garrett_Singer, Will_Law, Steve_Morris, Greg_Feedman, Tatsuya_Igarashi, Kazuhiro_Hoya, Nigel_Megitt, Narifumi_Iwamoto, John_Luther, Peter_Thatcher

Regrets

Chair
    Chris, Igarashi

Scribe
    kaz, cpn

Contents
    Topics
        Introduction
        Workshop on Web Games
        Web Games
        Threading
        Web Assembly
        3D rendering
        Asset loading
        Accessibility
        Audio
        Gamepad support
        Inputs
        Streaming
        Hosted apps
        Discoverability and Monetisation
        Next steps
        TPAC breakouts
        Discussion
        TPAC F2F
        AOB
        Conclusion
    Summary of Action Items
    Summary of Resolutions

<kaz> scribenick: kaz

<kaz> http://media.w3.org/2019/09/meig-2019-09-03.mp4 Recorded video from this call

# Introduction

Chris: Can I check if it's ok to record the call as usual?

(no objection)

(kaz starts recording on WebEx)

# Workshop on Web Games

<cpn> scribenick: cpn

<kaz> https://www.w3.org/2019/Talks/fd-games-workshop/report.html Francois' slides

Francois: The goal is to report from the workshop in June around web games.
.... W3C runs workshops as a way to investigate a new space or problem, as a starting point for further discussion.
.... We looked at web games as a whole, how web technology can be used to develop web games today, and what new technologies could be useful.
.... I'll report on the outcomes from the event.
.... There's a published report online.

-> https://www.w3.org/2018/12/games-workshop/report.html Games workshop report

Francois: I'll skim over the less relevant topics from our group's perspective.
.... There are some use cases that are related, some use cases from gaming are also useful from a streaming perspective.

# Web Games

Francois: We had a previous workshop on games in 2011, wanted to check on progress. Lots of new technologies emerged and standardised.
.... Web games create new paradigms not in the native world. There's no install, open a URL and start the experience.
.... Native games involve possibly large installation for all the assets.
.... WebAssembly is very popular, allows cross compilation of games from C++ or Rust into a package that can be run in the browser.
.... The interaction with web APIs is an interesting question here.
.... Recent announcements around cloud gaming. The game is run in the cloud on high-end servers with plenty of CPU, GPU, memory power.
.... The result is streamed to an end device, can be lower-end. Provided the client can send input events to the cloud, can provide gaming experiences without requiring features on the client.
.... New environments such as Facebook Instant games etc, a hosted environment where web technologies can be used.

# Threading

Francois: Useful for 3D rendering, but it was highlighted a specific need. When you want to render 3D content, or 3D media rendering, there's a scene graph.
.... Before you can send commands to the GPU to render, you need to process the graph in various ways, e.g., frustrum culling. This needs to happen on the CPU not the GPU.
.... It can be parallelised, but hard to do on the web, as Workers use message passing, they can share unstructured memory using SharedArrayBuffer.
.... This was removed from browsers after Spectre and Meltdown.
.... It's coming back, should be supported by all browsers.
.... Can't share a structured object with the worker, must serialize and then deserialize it.
.... Doing that 60 or 100 times per second doesn't work.
.... So a way to share structured data across threads is needed. Requires going deep into JavaScript-land, which is a single threaded model.
.... It's a need that was pushed by many people at the workshop.
.... Another need is that you can currently only start a worker via a file. It would be much more useful to be able to start workers using a function that inherits the calling scope.

<kaz> scribenick: kaz

Chris: What was the conclusion from the workshop, is the fact that you need a separate JS file a design constraint, or could it be made to work with a function too?

<cpn> scribenick: cpn

Francois: I believe it was done this was as it was easier. For a worker to take a function, you need to specify what gets inherited, e.g., shared memory, so it's not as easy as adding a new method to Workers.
.... But it's not impossible.

# Web Assembly

Francois: Another topic discussed was WebAssembly, the main issue that could be improved from a game perspective is calling Web APIs from WASM code is inefficient right now.
.... Consider a game engine that uses C or C++ such as Unity or Unreal Engine for example. It's hard to integrate with web APIs efficiently from the C++ code.
.... Another problem is around debugging. Once your C++ code is compiled to WASM, it's hard to debug the C++ that's running in the browser as WASM.
.... Luke Wagner presented the Web Assembly Bindings document. it's now called the Interface Types proposal. I won't go into details here.
.... This would make calling web APIs more efficient.

https://github.com/WebAssembly/interface-types/blob/master/proposals/interface-types/Explainer.md

Francois: There's also DWARF for Web Assembly debugging, using usual toolchains.

https://yurydelendik.github.io/webassembly-dwarf/

# 3D rendering

Francois: On the web, this is not on par with native. WebGL is old, misses functions that would improve rendering, and is less performant.
.... There are different texture formats supported by chipsets. The WebGPU API provides a common abstraction over different hardware.
.... There's an active CG working on this, will transition to WG. One issue before that can happen is choice of shading language.
.... Two main contenders, and some disagreement on which to choose, so it's an ongoing discussion.
.... A presentation from the Khronos Group on glTF. There's a discussion there on compressed texture transmission format, a common format that can be supported across platforms.
.... This is good news for developers, to avoid having to ship different texture formats for their models.
.... Should there be a 3D scene element on the web? It's been in the air for years, without leading to anything.
.... The workshop helped trigger discussion in the Immersive Web CG on glTF.

-> https://github.com/immersive-web/proposals/issues/52 glTF discussion

Francois: For accesibility of 3D scenes, need semantic information, there's joint work that could happen between Khronos Group and W3C.

# Asset loading

Francois: Web games are no-install experience. But still need to load assets before playing. 20% of players give up if there's a 10 megabyte download.
.... There's a need to address the loading process differently on the web.
.... Storing assets locally, but restrictions make it hard to store permanently on the device, e.g., high scores, vs a cache for game assets that could be downloaded again if needed.
.... Discussion at the workshop suggested not much needs to change for the loading experience. It's more about educating games developers about downloading things in the background, and postponing downloads.
.... For storage, an idea is to allow multiple buckets per origin. A difficulty is how to make the user aware of what's going on, what's permanent and what isn't.

# Accessibility

Francois: Games are meant to be challenging, so games are inaccessible by definition in some way. There's no one recipe to follow for accessibility in games. The technique would vary from game to game.
.... Some questions remain, e.g., how to provide semantic information to assistive technology when there's no DOM, i.e., when using which is an opaque set of pixels.
.... Problematic when the code is not JavaScript, cross compiled from C++, how to provide the semantic information from C++ code?

Kaz: There's standardisation work on WoT to extend the capability of web applications to devices such as Amazon Alexa, could be of interest.

<Zakim> kaz, you wanted to suggest we think about WoT as a possible tool for accessibility

Francois: We discussed the devices that people with motor impairment could be using, accessible input devices, what are the consequences for game design?
.... There's an incubation in WICG called the Accessibiltiy Object Model (AOM), it's a way to expose semantic information when there's no DOM.

-> https://wicg.github.io/aom/spec/ AOM spec

Francois: Combined with the WebIDL binding proposal, could be a way to integrate with C++. The two things in combination could be useful to address accessibility questions.
.... There are accessibilityy guidelines available, presented by Ian Hamilton and Matthew from The Paciello Group

-> http://gameaccessibilityguidelines.com/ Game accessibility guidelines

Kaz: Also, personal and privacy information. It might make sense to have some joint discussion at TPAC on these related topics.

Francois: Matthew is going to follow up in the APA WG, this is a logical place to discuss.

# Audio

Francois: There are lots of C and C++ audio libraries, professional code, used in native games for audio effects etc.
.... This code is hard to port to the web, because the Web Audio model doesn't work well in this case. There's a lack of a low level audio input/output API.
.... To support this, need better threading support, and there's a proposal in the Audio CG from Google: Audio Device Client for low level I/O
.... This would help with porting of existing code to the web

-> https://github.com/WebAudio/web-audio-cg/blob/master/audio-device-client/explainer.md Audio Device Client explainer

# Gamepad support

Francois: The Gamepad API has not made progress recently. Buttons may be exposed differently between browsers, so there's a combinatorial explosion to understand which button was pressed.
.... Controllers have modern features, touch surfaces, light indicators. You cloud make the gamepad blink the lights to indicate which one to use.
.... Also haptics support. These modern features link to the immersive world, there are advanced VR controllers.
.... Want to finalise the first version of the Gamepad API, close the exsting issues, then start work on a v2 API to support the modern features.
Inputs

Francois: For cloud gaming, need to send inputs to the cloud for rendering. Input events are aligned to rAF for scheduling.
.... This isn't suitable for cloud gaming. All input events need to go through the main thread, leads to congestion issues, helpful to process some events in a worker.
.... Mouse events are accelerating and in FPS games, it makes it hard to make precise movements.
.... There's a Google proposal for Input to Workers and Worklets.

-> https://wicg.github.io/input-for-workers/ Input to Workers and Worklets

Francois: Also raw events, to get events as soon as possible. Evaluating whether the cost is worth it.
.... Also unaccelerated mouse events.

# Streaming

Francois: Problem with WebRTC solution is that you can't fine tune the encoding/decoding parameters, also congestion control.
.... Two ideas discussed, WebTransport based on QUIC, a UDP version of Web Sockets. Allows developers to write their own congestion control.
.... Incubated in WICG. Also the WebCodecs proposal, not yet in WICG, needs support. This would expose the encoding and decoding capabilities of the browser.

-> https://wicg.github.io/web-transport/ Web Transport
-> https://github.com/pthatcherg/web-codecs/blob/master/explainer.md Web Codecs explainer

# Hosted apps

Francois: Some APIs might not be available in hosted environemnts, and some APIs are specific to hosted environments.
.... Different companies have different hosted environments. Could be beneficial to converge these.

# Discoverability and Monetisation

Francois: Easy for things to disappear on the web. Metadata formats, such as schema.org, for game capabilities: categories, input controllers, offline support, what monetization schemes does it implement?
.... Monetization on the web is broader than games. Watching ads to add features to the game.
.... Micro-transaction proposal in WICG.

-> https://github.com/adrianhopebailie/web-monetization/blob/master/explainer.md Web Monetization explainer

Next steps

Francois: Agreement at the workshop for further activities, could be a new IG. Could it fit in scope for the M&E IG, maybe as a new TF?
.... It's a different topic for this IG. Is there enough support for some new activity?

# TPAC breakouts

Francois: There's one on mini-apps, hosted environments. Also WebTransport and WebCodecs, and WebGPU. A 3D HTML element, from someone not at the workshop
.... and the Input for Workers/Worklets

-> https://w3c.github.io/tpac-breakouts/sessions.html TPAC 2019 breakouts

Francois: The Audio WG will meet, and WebApps WG will talk about the Gamepad API

<Barbara_H> Latency is a key challenge. What about a discussion at the Networking IG on latency?

Barbara: So latency is key. Could discuss at the Web and Networks IG on latency and the impact on gaming?

Francois: It could be, yes. WebTransport would be good to discuss there, also other solutions, especially thinking about 5G. Good point, yes.

# Discussion

Barbara: Is it this group, or another group? High-end gaming will push the graphics aspects. There's also mainstream gaming happening.
.... We'd need to look at the gaming model segmentation. It's OK to start by grouping them together, but the needs of a high end dedicated gamers vs casual gamers will start to grow.
.... Looking at the business model, high-end dedicated gaming segment will grow, but they have different expectations to move to the web.
.... The casual gaming segment is the growing market. Different approach for established market moving to the web vs an emerging market.

Francois: You're right. We struggled to get AAA producers to the workshop, although we had the main game engines, but that's more technical.
.... They see the web as an important platform, but they're not prioritising it.
.... Then there are the indie game developers, more keen on web technology, for reach.
.... And in the middle are the game platforms, more low end, but sharing high end needs.
.... Happy to take suggestions.

Barbara: About machine learning, games will be impacted in the future, it's a common theme.
.... Another thing to monitor going forward, intersection with cloud gaming.

Francois: We left machine learning out of scope, we'll have a future workshop on that.

Will: WebRTC was passed over in terms of improvements. Why not fix some of those issues with WebRTC vs inventing a new distribution scheme? It has a huge penetration now, allowing customisable FEC or congestion control would be good improvements.

Francois: The WebTransport proposal can be plugged into WebRTC. If you expose congestion control to WebRTC, you end up with a similar solution.

Peter: The work on WebCodec and WebTransport originated from WebRTC, trying to provide more control to the application. It grew out of the WebRTC WG because of the applications outside of RTC that were finding these new APIs useful.

<kaz> scribenick: kaz

Chris: I think this is a good topic for us to follow up on. We have two breakout sessions at TPAC.
.... Should we schedule something after TPAC to go into more detail? Not everyone will be at TPAC.
.... We can follow up offline about scheduling a call on Web Transport and Web Codec.

<scribe> ACTION: Chris to arrange IG call on Web Codecs and Web Transport

<cpn> scribenick: cpn

Rob: To Barbara's comments, there's an open source animation package called Blender. A great idea they had was to create open projects, films or animations that are community sourced.
.... An open project for a game could be a good idea, to prove out the technology. So while you're building the APIs, you're also building a game and involving the community. It would showcase the benefits, highlight any weaknesses or problems, get feedback and buy-in from the community during the develoment.
.... At the end you'd have the system, with examples of how to use it. It would help with the hard-core vs casual argument, it would become clear which areas it would support well and which might be problematic, e.g., can you produce a 3D shooter due to latency issues?
.... Would save effort on things that wouldn't work.

<RobSmith> https://cloud.blender.org/open-projects

<kaz> scribenick: kaz

Chris: What support is there from participants at the workshop? There was a lot of interest, but what about taking these topics forward? Is there a push for creating a TF or IG?

Francois: Have discussed with some of the participants. Some are positive about the idea, but none were really willing to drive it.
.... We need to find one or two organisations who would drive the discussion, otherwise it's hard to create a group and do something useful.

<cpn> scribenick: cpn

Francois: We created a games CG after the first workshop, but it gradually died. So we need people to be able to commit to running the effort.
.... I'm talking mainly with people at Facebook, they have a good over-arching perspective, could be a good first company with an incentive to push an activity, no commitment yet.
.... If you're interested to help, please get in touch.
.... An IG is harder to create, needs a scope and charter, getting member support. A TF is more lightweight, is it a good idea to create in the M&E IG? Some of the topics are foreign to this group until now.

<Zakim> kaz, you wanted to suggest we think about WoT, DID (and possibly DAS) as well for TPAC

Kaz: I agree with other comments made here, and want to suggest having joint discussion with WoT, DID, DAS, etc. at TPAC

<kaz> scribenick: kaz

Chris: TPAC is a good opportunity so discuss creating a group.
.... I think the key is having people to lead the activity, and following that we can talk about the venue, either this IG or a new group.
.... I'm very happy to participate in those conversations.

Francois: That makes sense.

# TPAC F2F

Chris: Our IG meeting will be on Monday, Sep. 16. If you can't be there, we'll have a Webex open to join remotely.
.... The agenda is mostly finalised.

-> https://www.w3.org/2011/webtv/wiki/Face_to_face_meeting_during_TPAC_2019#Agenda_Monday_16_September_2019 TPAC agenda

Chris: We have a joint meeting with Second Screen WG/CG.
.... We'll discuss accessibility and captioning in VR and 360 video.
.... We have open discussion during afternoon, looking at the next steps for media on the web.
.... Things like Web Codecs and Web Transport feed into this.
.... Now that we have a Media WG, there's an opportunity for the IG to look at its contribution to those APIs, but also to look beyond that to use cases that drive further development of media support on the web.
.... If you're coming, I look forward to meeting you there.
.... If you can't make the meeting in person, you're welcome to join remotely.
.... We have some media related breakouts on Wednesday, a lot of interesting topics.
.... This should be a good meeting.

<tidoust> Proposed breakout sessions at TPAC

# AOB

(none)

# Conclusion

Chris: Thanks for joining, and thank you for reporting, Francois.

Kaz: Next call on Oct. 8?

Chris: Not sure, I'll be away. I'll discuss with the IG co-chairs.

[adjourned]
Summary of Action Items
[NEW] ACTION: Chris to arrange IG call on Web Codecs and Web Transport

Summary of Resolutions
[End of minutes]
Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)
$Date: 2019/09/05 15:22:58 $