Re: GamepadObserver (ie. MutationObserver + Gamepad)

Sorry for the long-delayed reply.  The "tablets" thread reminded me about
this mail, which got lost in the black hole known as "drafts"...

On Mon, May 7, 2012 at 2:33 PM, Scott Graham <scottmg@chromium.org> wrote:

> - As you point out, the 360 controller is by far the most common PC
> gamepad, and on its native platforms its standard API XInput is
> exclusively polling based [4]. That doesn't mean we must or even
> should follow that design, but it does mean that even if we design a
> bunch of fancier/cleverer APIs we're still going to be largely
> constrained by that API. That is, there's not going to be any extra
> information in events that are synthesized from polled data. [5]
>

Not exactly, since you can poll devices natively with these kinds of APIs
much more quickly (without being excessively wasteful) than you can in JS.
 Polling at 120Hz (or even 1000Hz) natively, in a well-tuned thread, is
much more realistic than doing it in JS.  You can also do it in a
high-priority or realtime thread, so timestamps are as accurate as possible.

 > - Native deadzone handling (by the OS or internally to the device) is the
> > only way to correctly handle deadzones when you don't have intimate
> > knowledge of the hardware.  It should be explicitly (if non-normatively)
> > noted that native deadzone handling should be used when available.
>
> This has been discussed before, and it's certainly important. The one
> complexity that needs to be solved is handling 1D vs. 2D dead zones
> for axes. That is, should the X and Y be deadzoned as independent 1D
> axes, or should they be deadzoned as a 2D vector. I guess this could
> either be left to non-normative discussion and hope the
> implementations do something sensible, or additional API surface will
> be needed for a full solution. (I would probably lean towards
> non-normative, and ignore the separate 1D axes problem, myself.)
>

Some hardware does deadzoning internally.  Also, the type of deadzoning may
depend on the device; the correct answer may be different for gamepad
thumbsticks and joysticks, for example, and devices with different amounts
of slop need dead zones tuned differently.

I think this should be entirely up to the implementation.  However, it may
be worth a note that it *is* intended that deadzoning be handled either by
the browser or somewhere lower down the stack, and not by web developers
(up the stack).

 > I'd suggest a halfway point between polling and events: a function to
> > retrieve a list of device changes since the last call, and an event
> fired on
> > the object the first time a new change is made.  For example,
> >
> > var gamepad = window.openGamepad(0);
> > gamepad.addEventListener("input", function(e) {
> >     var changes = gamepad.readChanges();
> > }, false);
> >
> > with changes being an array of objects, each object describing a
> timestamped
> > change of state, eg:
> >
> > changes = [
> >     {
> >         button: 0,
> >         state: 1.0,
> >         lastState: 0.85,
> >         timestamp: 1336102719319
> >     }
> > ]
> >
> > This allows using polling if your application is based on
> > requestAnimationFrame, or events if you want to be told when there's
> > something to read.  There isn't an excess of events dispatched, because
> the
> > event is only dispatched once per call to readChanges; if you only read
> > changes once in a while you'll only receive the one event.  It also
> solves
> > all of the above problems with a most-recent-state-only interface.
>
> I think it's a loss to not have a full most-recent-state available.
>

This isn't mutually exclusive with a separate getCurrentState() method.

 Many common styles of applications (rAF+games) would have to add a
> bunch of boilerplate that then merges these events to maintain the
> current state (especially awkward on XInput since the browser will be
> splitting them out of most-recent-state!). I don't see a lot of
> benefit to splitting each individual axis/button change out into a
> separate object. Is there a reason you prefer that way?
>

It seems natural to me.  You can detect the order of events, their relative
timing, and you don't have to analyze a data structure to determine which
piece changed.  If all you care about is the most recent state, then use
something like getCurrentState().

When readChanges() happens, I guess we'd need to consider some sort of
> staleness? What happens if a client only reads once every second? Or
> one every 5 minutes?
>

Implementations should definitely be able to cap on the amount of stored
data.

For the most part, events can simply be dropped from the beginning of the
queue.  However, button release events should never be dropped unless
there's another button depress event later in the buffer.  Otherwise,
buttons would get "stuck" to the application's view.


 > The timestamp should be as accurate as possible to when the event
> actually
> > happened, in the clock used by Date.now(), so you can tell how long ago
> the
> > event happened.  Accurate timestamps are important for games that require
> > fine-grained timing; 2D fighters and rhythm games can make use of this,
> for
> > example (they may render at 60 FPS but perform game logic at a much
> higher
> > rate).
>
> Definitely it should be as accurate as possible. I was assuming it
> would be the same as window.performance.now() [1]?
>

That's fine, too (assuming that API gets implemented).



> > Any serious gamepad API must not ignore the realities of the most common
> > joystick layouts, most importantly the 360 controller (far and away the
> most
> > common PC gamepad today).  The API needs to give recommended mappings
> from
> > buttons and axes to the parts of those real-world controllers, so users
> > don't have to configure devices manually.  This is absolutely critical;
> > every native PC game that supports gamepads now "just works" for the 360
> > controller, requiring no configuration.  (For years joystick APIs on PCs
> > tried to pretend that joysticks could be boiled down to a bunch of
> abstract
> > axes and buttons.  The user experience that results in is abysmal.)
>
> In the prototype I wrote in Chromium/WebKit, I've used the 'id' field
> as a lightweight version of trying to make the data somewhat usable in
> the real world. For recognized gamepads (a handful of common pad-style
> devices), the order of buttons and axes is normalized to a canonical
> ordering [6], *and* the string 'STANDARD GAMEPAD' is included in the
> 'id' field. This is only done when no information is lost by doing the
> remapping (that is, no buttons or axes are ever removed).
>

However, this needs to go both ways: if it can't fill out the standard
layout completely, it should use a different layout.  Otherwise, it doesn't
work: games will map functions to buttons and the user won't have any way
to push them.

On the other hand, we don't to exclude the ability to use other
> devices. I have had people contact me about using 3D mice [2] and foot
> pedals [3] via this API, and it would seem odd not support those
> similar uses. So, having the raw axis+button arrays available for more
> custom devices and applications seems important too.
>

Sure.  If the UA and/or spec has no notion of how to map a device, then it
should at least fall back on "input soup" so developers can still do
something with it.

 > If the gamepad model is known, it should also be explicitly exposed, so
> > applications can show help texts which match the actual buttons on the
> > gamepad (with a registry somewhere--which could just be a wiki
> > page--describing the string to product mappings).  Every native PC game
> also
> > now does this, finally catching up to what console games have done for
> > decades.  If a web API for gamepads doesn't do this, it'll be years
> behind
> > everything else.
>
> This was the intention of the 'id' field. The combination of 'STANDARD
> GAMEPAD' and including the USB vendor and product id, as well as other
> driver name information gets us some of the way there. It's currently
> ad-hoc though, and we probably want to clarify this as we understand
> better how authors want to use it.
>

Browser vendors have consistently shied away from exposing things like
hardware IDs, though.

> Additionally, a more involved system of input profiles would be needed for
> > practical forward-compatibility, eg. so applications can say "this is a
> list
> > of device types I understand" and UAs can expose the device as the
> nearest
> > match.  This allows old applications to work better with devices they
> don't
> > know about, without requiring the user to manually bind buttons and
> axes.  I
> > can't stress how important it is to not require the user to configure
> every
> > game manually; it's simply no longer acceptable.
>
> This gets very complex to specify, I think. We had previously pushed
> this towards a content library. The standard style/layout would help
> alleviate this somewhat though.
>

The problem with a library is that people will mirror it for their
application, and then not update it.  It also gives implementors no room
for improvement.  When a new device is released, instead of a few browsers
updating (and users having the option of using the browser that supports
their device best), thousands of web pages will need to do it.  That'll
give very spotty support for anything but the most common device.

I'll brainstorm this a bit, and if I come up with something that seems
plausible I'll post it separately.

[1] http://www.w3.org/TR/hr-time/
>

Please link to editor's drafts; TR specs are usually obsolete, which leads
to confusion.
http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/HighResolutionTime/Overview.html

[5] My understanding of USB is limited to a few quick searches, but as
> I read it, the default USB hardware polling rate is only 125Hz = 8ms
> anyway. Anyone familiar with hardware know if that applies to
> many/most gamepads?
>

It's been a long time since I've dealt with that level, but my
understanding is this is only the polling rate; devices can send interrupts
without any delay.

Also, it's possible to implement controllers which give timing and ordering
information, without necessarily sending more messages.  I've implemented
firmware for game controllers with this as a design goal in the past; it
preserved ordering and gave event timestamps with an accuracy of about 1ms,
despite having an RF protocol that polled at a much lower rate.

-- 
Glenn Maynard