Re: [gamepad] Allow standard-mapped devices to have extra buttons or axes & allow multiple mappings per device

Hi,

I agree with Chad's suggestions: the spec should be expanded to expose
more than the core set of axes & buttons, and richer metadata would be
extremely useful to developers.

I've been using the gamepad API (as shipped in Chrome, at least - I
haven't had time to adapt to Firefox's implementation, because I used
gamepad.js and it's a broken pile...) for a while now, in a few
shipped JSIL-based game ports. My experiences & opinions inform the
following.

I apologize if this is a bit hard to read, by the way - all this
crunch time for the game I'm working on has fried my brain a bit. :-)

To try and express my PoV on this (previously stated to WG
members/participants, but not directly on this list - sorry, too much
on my plate to add another workgroup to my subscriptions):

I think best-effort mapping to 'standard' layouts is a great idea. I
have some objections to how it is done, but it is an improvement over
what we had before. My main concern is that the current design for
remapping sets us up for a descent into a quagmire that can't easily
be fixed. The scenario I anticipate happening:

Gamepads only have one standard mapping, supported by the vast
majority of available controllers (let's just say 360 pads and
dualshock 3/4 controllers, since those put together probably cover at
least 90% of the market). If you expand the remapping to all XInput
devices, that widens it even further - though this is problematic in a
way I will get to later.

If there is one standard mapping, and it covers most users, this means
the natural path for developers is to only support the standard
mapping. They won't add keybinding/button remapping, they won't add
support for multiple layouts, they might not even add support for
choosing which of your gamepads you want to use. This is not an
unrealistic fear: It is the status quo in virtually all console games
and in many PC games. In some cases you can rebind by editing .ini
files, but that is out of the question for web games since their
configuration typically lies in not-user-editable JS on the server or
in blobs stored somewhere (localStorage, indexed db, cookies) that are
not easily user-editable. This 'developers only support the easiest
thing' future is relatively likely and is actually worse than the
previous status quo of raw-controllers-only.

I think the solution to this is simple: Ensure that there is more than
one standard mapping. Find a way to make sure that developers have to
do at least a basic amount of effort to handle things correctly. If
the simple hack isn't sufficient for wider audiences, they will have
an incentive to do a good job. This doesn't have to be a punitive
measure - there are diverse controllers out there that people might
want to use, from standard gamepads to racing wheels to flight sticks.
Something as simple as supporting the additional inputs of the
dualshock 3 & 4 is a good rationale here (they both have gyros &
accelerometers; the 4 has a touchpad and touchpanel button.)

When the common case is slightly more complicated, it forces
developers to at least *understand* the breadth of the problem and
make rational choices. It also encourages the development of
high-quality, userspace JS wrappers/abstraction layers, so fixes &
improvements can happen in those libraries. I would argue that the
success of XInput is not due to its excessive, somewhat-detrimental
'least common denominator' design, but because it aggressively
targeted & fixed the most common problems with DirectInput.

OK, so, there's my rationale for why the current approach to mapping
should change. Now to dive in deeper to some other issues I see with
the gamepad API.

My first objection to the design & implementation is that it is based
on passive enumeration & device acquisition. You have to poll to find
out what gamepads there are and interact with them. This obviously
produces the threat of fingerprinting, and to mitigate fingerprinting
we have the *miserable* constraint where you have to push a button to
make your controller show up. This changes the ordinal identifiers of
gamepads from session to session, and - even worse - means that a
controller cannot be activated by analog stick inputs, which is a real
problem for games that start with a menu (like two of my major ports).
This defies user expectations and is just plain confusing. Polling an
actual device to get its state is perfectly reasonable, but polling
the overall state of your input devices is not so. Device availability
is something that should be event-based (it is at a driver level
anyway) and it should be exposed to content as events.

The fingerprinting problem is addressed with the solution I proposed
here: Content should *request* the input it needs, at which point the
useragent is able to figure out how to provide it. To mitigate
fingerprinting concerns, this can be done based on user consent, and
the input presented to the content can end up fully sanitized, because
the user content has no need to identify devices and make choices
about how to map inputs. The request can clearly specify the
*semantic* inputs the game requires, like an 'accept button', instead
of just identifying buttons by ordinal or some rigid layout with
buttons called things like 'a' and 'x'. Semantic requests can easily
be an optional thing, at which point the common case is roughly
identical to the present. Semantic requests for buttons also mean that
the game has a user-friendly name to display for each button, and that
name can be properly localized.

This leads into my second objection: I think it is unwise to straddle
the boundary between useragent & content the way the current approach
does. The browser makes a 'best effort' at remapping, but there is no
real facility for the content to figure out how good the 'best effort'
was, or to opt into getting the raw state so they can do a better job.
The fact that a 'best effort' will not always be possible means that a
bunch of the work here - arguably some of the most important work - is
shoved off into content, but the common case is easy enough that
content may not cover the less common cases. This really sucks.

I think any solution that can't exist entirely in the useragent must
exist in content. I argue that the correct approach to remapping is to
do it in userspace JS. The fact that remapping has already shipped
prevents this, but perhaps there is some sort of compromise that can
be reached - on first use of the gamepad api, perhaps, a blessed
userspace JS library is loaded that provides remapping, and there is a
trivial way for content to request that a content-provided version of
the library is used instead. This would end up working sort of like a
polyfill in reverse, where if you've loaded a polyfill the browser
does not trample over it.

I think a relatively ideal - not perfect, realistic - solution in
content can also solve problems that the browser can't solve by
itself. In detail:
* The blessed polyfill can be officially hosted on a
community-controlled domain - perhaps shepherded by a known good actor
like Mozilla or Google - that acts as a canonical location for the
gamepad mapping support.
* As a result, the community domain can act as a storage repository
for the user's button mappings & other information, perhaps only
hosting a simple script that exposes this shared data to individual
games and applications. This storage doesn't have to exist in the
cloud - the domain just acts as an origin any localStorage or
indexedDB data is tied to, so all content can access it.
* Given the shared storage repository, the burden of custom button
mapping & device identification/remapping can be entirely offloaded to
content while simultaneously improving the experience for users.
* The blessed library can provide a user-friendly API & interface that
covers all the corner cases - a good UI that shows the inputs the game
wants and allows the user to map them, along with an easy 'default'
that shows you the best-fit mapping so you can decide whether to use
it. This means that a user who doesn't want the best-fit mapping -
they want to invert y, they want to rearrange the buttons because they
have a disability that makes the standard layout difficult - can
easily opt out of it.

I believe my proposed approach addresses all the problems the current
gamepad API solves, while offloading a lot of the burden from the
browser developer to content. In addition, it solves some problems the
current API does not. The ability to save and customize bindings on a
global basis instead of on a per-game basis would be a HUGE usability
improvement for web games, and more importantly it would be an
advantage traditional PC games can't offer.

In practice, for developers this would mean a new API sitting on top
of the existing gamepad API, where instead of polling for gamepads,
you ask for one or more virtualized gamepads that fit your game's
requirements. If the user has already opted into it before, you just
instantly get the devices you want, if they're available. No need to
cope with fingerprinting protections. If the user has existing
bindings for your game or another game with the same semantic
requirements, they have no need to bind inputs again. If you need a
new 'standard' layout like a flightstick or a racing wheel, you ask
for that instead. The part where you've already got a gamepad remains
the same: None of that API has to change. Just the part where you
acquire devices. This means the existing implementation (warts and
all) can remain and the new, ideal design can replace it side-by-side.

I'm aware that my comments here & my proposal are rather complicated
so I'd be happy to explain them in more detail.


Full disclosure: I was involved in the initial prototyping & spec work
for the gamepad API that happened at Mozilla, up until shortly before
Google's shipped prototype API became the de-facto standard. I'm
slightly bitter, if only because the request-oriented approach to
acquiring gamepads is what I proposed originally and fought hard to
make happen on our side. I'm a bit sad that the current approach
shipped because while it solves important problems, it sets us up for
some pain in the future.

On 13 October 2014 17:58, Chad Austin <caustin@gmail.com> wrote:
> On Mon, Oct 13, 2014 at 3:55 PM, Brandon Jones <bajones@google.com> wrote:
>>
>> Re: change #1, with standard gamepad mappings today (at least in Chrome)
>> we map any buttons that don't correspond to the official standard mapping to
>> button[17] and up. This, of course, depends on which buttons are actually
>> visible to the browser (many devices expose buttons that can't be "seen" by
>> normal applications, like the home button on Xbox controllers). A good
>> example is the Touchpad on the PS4 controller. We can't see actual touch
>> inputs without writing a custom driver, but we detect clicks on the touchpad
>> and report them as button[17]. The same would go for axes, but I've yet to
>> encounter a standard-looking gamepad that exposed > four axes worth of
>> usable data. Perhaps the spec should be slightly reworded in this regard,
>> since it currently states that "The standard gamepad has 4 axes, and up to
>> 17 buttons", which could be read as explicitly disallowing buttons or axes
>> beyond that.
>
>
> Great!  Then I agree - the behavior of Chrome then doesn't match the exact
> wording of the spec.  The spec should be updated to reflect reality.
>
>> I'm not very supportive of suggestion #2 since we don't actually HAVE any
>> other mappings defined (joystick/flightstick would be the next most natural
>> candidate, I think, but they can vary significantly). If we did, I'm not
>> sure how you would reasonably "stack" mappings without breaking an awful lot
>> of code. The "mappings" field would have to be either space/comma delimited
>> (which breaks things) or turned into an array (which breaks things) or you
>> would expose the same device twice under different mappings (which is really
>> weird and probably breaks things.) Beyond that, there's no way that the
>> layouts of two mapping types could every overlay completely cleanly, and we
>> really shouldn't be claiming that something that looks like a racing wheel
>> is actually a "standard gamepad" anyway. I just can't see how it would work.
>
>
> What's the status of the gamepad API?  It reads like it's in a draft state,
> yet it has some implementations in the wild...  Is it de facto frozen?
>
> Assuming it's not de facto frozen -- and temporarily sidestepping the
> problem of maintaining compatibility with existing implementations -- the
> issue with the mapping API _as written today_ is that there's only one
> standard mapping, meaning that if you plug in any non-XInput-style device
> that has a d-pad, you'll need the customer to specify their own bindings.
> "Your device is nonstandard.  Press left.  Press up.  Press right.  Press
> down.  Press OK."
>
>> Beyond that, there's no way that the layouts of two mapping types could
>> every overlay completely cleanly, and we really shouldn't be claiming that
>> something that looks like a racing wheel is actually a "standard gamepad"
>> anyway. I just can't see how it would work.
>
>
> The concern that two mappings that each specify a directional pad would not
> have overlapping button IDs is valid.
>
> Fundamentally, what I'd like is access to a richer form of per-button and
> per-axis metadata than is provided by the API today.  The "standard mapping"
> API provided today is rigid and limiting.  However, I don't have time to
> come up with a proposal right now - perhaps coming soon.
>
> +Katelyn as she expressed similar concerns.
>
> Thanks,
> Chad

Received on Wednesday, 15 October 2014 07:44:11 UTC