RE: [DOM3Events] InputDevice API sketch

Sure, let’s first focus on uncontroversial bits like “what are the usual behaviors of this device” before trying to define those default behaviors accurately. I was just trying to be concrete by citing concrete examples, as I feared I would be too abstract otherwise. 

 

A good outcome for this thread would try to identify such behaviors/behavioural properties. I can think of:

 

-          isFine/isCoarse (already ok)

-          hasHover (already ok)

-          hasContacts/maxContacts (mostly ok)

-          hasButtons

-          hasMouseWheel

-           

-          usesTapAndHold

-          usesDirectManipulation

-          usesSecondaryButtons

-           

-          firesTouchEvents (already ok)

-          firesPointerEvents

-          firesGamepadEvents

-          firesMouseWheelEvents

-          firesKeyboardEvents

 

We should also make sure we can query a pointer device about all the properties which a Pointer Event (generated by this device) will feature

 

-          hasPointerPressure

-          hasPointerWidth

-          hasPointerHeight

-          hasPointerTilt

-          isPointerPersistent (the pointer never dies, like the mouse pointer)

 

If someone can think of anything else, feel free to append to the list.

 

Please note I’m not trying to make an “all of this or none of this” statement, I’m just trying to make sure we explore the InputDevice world sufficiently deep before we start settling ;-)

 

Best regards,

François

 

 

De : rbyers@google.com [mailto:rbyers@google.com] De la part de Rick Byers
Envoyé : jeudi 9 avril 2015 18:22
À : François REMY
Cc : DOM WG; public-pointer-events@w3.org; Gary Kacmarcik (Кошмарчик); Domenic Denicola; Mustaq Ahmed
Objet : Re: [DOM3Events] InputDevice API sketch

 

 

 

On Thu, Apr 9, 2015 at 8:44 AM, François REMY <francois.remy.dev@outlook.com <mailto:francois.remy.dev@outlook.com> > wrote:

Wow, thank you very much for this well-balanced reply. I really appreciate the time you must have taken to write it, not even counting the time you probably took to have a look at the proposal in itself. ❤

 

To be completely honest, I somehow knew this wasn’t the right place to make this proposal because, to get there, the very first step is to get a new Input Device Driver standards which allows for (at least some of) such features to be exposed at the OS level. The thing is: I don’t know anyone from that field, and even if some people there agreed to work on it, it’s definitely going to take years to get to a point where the design I propose could give us any yield. I recognize that.

 

The real purpose of my intervention was mostly to expose real use-cases I’ve seen working (like Touch Input on a trackpad, on a Touch Mouse, via a Depth Camera, what happens when you have two mices connected, etc) and make sure we have them in mind when making additional incremental changes to the platform. 

 

In short: I want to make sure we adopt when it’s possible a cross-device vision and not a per-device vision (like most of the API have at this point).

 

An example of this philosophy would be to expose properties like “inputDevice.defaultLongPress”, “defaultLongPress.timeout”, “defaultLongPress.maxIgnoredMoveDistance” etc which allows a PointerEvent listener to have the right “tap&hold” feeling for the Device+OS combination (if any is necessary) without having to test “is this device a touch device” or “is this device a mouse?”. Ditto for scroll/manipulation APIs. We want to know if a device is expected to trigger panning and (if yes) with which initial move threshold tolerance. Currently we only have the choice between a mouse which doesn’t need any of this and a touch screen which does. A new input device will have to comply with either one of those contracts. #SadPanda

 

I hate very much the idea to have an event handler “customized” for each pointing device type, and not relying on device properties (and right now this is how a PointerEvent-consuming code looks like). That makes it hard to introduce new types of input devices. Let’s try avoid any new “Touch Input fires ontouchmove then onmousemove” fiasco with new input devices in the future.

 

To sum up, I would like to take advantage of the fact you propose to define InputDevices to, at the very least, expose behaviors & behavioral properties, and not device types (when it’s reasonably possible). 

 

This requires some more work, but I think this work is necessary to open doors to the future. Let’s not be too minimalist when we want to be realistic ;-)

 

Yes, I agree this is a good direction in general - have developers ask the thing they really want to know (does dragging this pointer cause scrolling), rather than infer it from some related property (is this pointer a touch).  I actually raised concerns about PointerEvent.pointerType during the original standardization (i.e. that no new device would really end up choosing a value other than the original 3 because it would break some subset of websites).  But we agreed we probably wanted to keep it for close compat with IE 10, and we could add the more generic properties I suggested as future devices became standardized.  This is why we have 'pointer' and 'hover' media queries instead of 'hasTouchscreen' and 'hasMouse'.  I consider this InputDevice API to be the next stage of that debate.

 

Each possible API will need independent debate though.  Eg. I personally don't think a "move threshold tolerance" API is a good idea - I believe Safari, for example, relies on an algorithm that's more complex than a simple bounding box.  For that particular use case we've instead discussed an API like TouchEvent.willTriggerScroll - so that for a given event you can see if it's about to cause a scroll to occur.  I'm not sure offhand how to map that idea to pointer events (which never block scrolling), but given a suitable concrete and important use case I'm confident we can.

 

That said, back compat is always going to be an issue that's impossible to avoid completely.  New interaction modes may need to differentiate between behavior needed for compatibility and the true behavior intended for code that is aware of the device type. We can do our best to minimize this with some upfront thought, but future-proofing is impossible and guessing too hard can add unnecessary complexity and confusion without ever actually achieving the goal of being entirely future-proof.

 

Let's not forget that what makes the web truly unique is the incredible reach, interoperability and lack of centralized control.  Dealing pragmatically with decades-long back-compat challenges is the (large) price we pay for that benefit.  Sure it would be nice to periodically deprecate all the old cruft and replace it with newly designed API surface (like native platforms tend to do in some way every ~5-10 years).  But to me that annoying cost is worth the huge benefit.  As you know, taking even the small leap here needed for Pointer Events is still far from being a platform success story, but we're trying ;-)

 

 

François

 

 

PS: More comments inline tagged with [REPLY] but those are less important

 

 

De : Rick Byers <mailto:rbyers@chromium.org> 
Envoyé : ‎jeudi‎ ‎9‎ ‎avril‎ ‎2015 ‎04‎:‎16
À : François REMY <mailto:francois.remy.dev@outlook.com> 
Cc : DOM WG <mailto:www-dom@w3.org> , public-pointer-events@w3.org <mailto:public-pointer-events@w3.org> , Gary Kacmarcik (Кошмарчик) <mailto:garykac@chromium.org> , Domenic Denicola <mailto:d@domenic.me> , Mustaq Ahmed <mailto:mustaq@chromium.org> 

 

Thanks for your detailed response François, and sorry it took me so long to reply.

 

I like the idea of provide more abstract input device information, and there are a number of nice properties in your design.  However it's hard for me to imagine a big highly-abstract API like this really being successfully deployed to the web platform.  It's not enough for a browser to just implement this API (as you say, mostly hard-coding it based on the less-abstract APIs provided by the host OS), but to be really valuable each browser would have to also do work per input device.  In practice I'm not sure we'd really get browser vendors excited enough to implement more than the most common devices.

 

[REPLY] I shall ship beer and cool input devices to all browsers vendors daily until either excitement arises or my money stock vanishes ;-)

 

I'm personally most interested in incremental approaches where we can add some small surface area but have the ability to efficiently iterate adding small feature after feature as the need arises (without being afraid to even approach the standards group due to the expected effort required).  I.e. after months of debate, let's see if we can get even my simple "does this device fire touch events" issue addressed by a W3C spec before we attempt something with orders of magnitude more abstraction and bikeshedding opportunities ;-).  I have low confidence that we could design an API in advance which will automatically work great for new devices without API changes (eg. even your highly generalized API is probably no better at supporting the new Mac force touch touchpad than pointer events would be).  So to me being able to iterate efficiently is more important that predicting all possible use cases in advance with a highly abstract API.

 

[REPLY] Yes, definitely. 

 

However, I think this just highlights the problem of the specialist scenarios being blocked on the standards consensus process and browser prioritization funnel (which - when it works at all - really works only for the most common use cases).  

 

[REPLY] +1

 

As long as the browsers need to add code to support each new type of input device (especially when new API surface is involved), we're always going to be playing catch-up (eg. imagine if leap motion couldn't successfully ship a device without convincing the Windows, Darin and Linux kernel teams to implement some feature, blocking on adequate deployment of updated OS versions).  Personally I believe the only practical path to the sort of fully generic device support utopia you're advocating for here is via simpler standard primitives.  Eg. perhaps with low-level web bluetooth and USB APIs you could implement the abstract API you're looking for here in a JS library?  Once a particular device/scenario gains popularity we can then justify investments to make that particular scenario better (eg. perhaps with a higher level API that can be implemented without bothering the user with a permissions prompt).

 

[REPLY] This means more special-cased code, though. At the very least, we should provide a way to fire consistent polyfilled Pointer/Touch events. What about PointerEvent.registerCustomPointer() which would return a free pointer id, and unregisterCustomPointer(id) which would free it?

 

Still I think there are other ways to make pragmatic incremental progress towards some of your goals.  

 

[REPLY] I hope so

 

Eg. we've talked about raw touchpad access in PEWG a few times, and it's something I'm moderately interested in seeing happen (eg. for handwriting recognition scenarios).  I believe Jacob (PE editor) and I agree that a small incremental extension to PointerEvents / pointer lock could address that use case - but we'd really need a champion for the use case to drive it in the WG (Microsoft doesn't generally have adequate touchpad APIs / hardware, Google cares most about phones, and Apple doesn't participate in PE).  If anyone is interested in helping push this (especially someone that would actually consume the API in a shipping product), public-pointer-events is the place to do so (I promise to help facilitate).

 

[REPLY] That won’t be me, but it’s great to see people are open to it.

 

So I think it's most important that we establish some minimally abstract framework that is simple while still flexible enough to cover the variety of most likely use cases.  I think PointerEvents is part of this, but we always knew we were missing some place to hang device query APIs off.  Hopefully a simple InputDevice API can give us the other half.  This still doesn't do much to help non-pointer devices but perhaps that's not critical enough to worry too much about - I don't know.

 

[REPLY] At least, this will make things better for what exists today, so it's definitely a win. Other input devices still have work to do before we can consider them standards, so it’s fine delaying.

 

Sorry not to be more positive - just trying to be realistic about how the web evolves in practice...

 

[REPLY] The contrary would have surprised me, to be honest. I used the “make people see your idea's big picture, to make it easier for them to understand it while micro-applied to smaller cases” strategy here. 

 

[REPLY] Thank you very much again for allocating some of your time to this.

 

Rick

 

 

 

 

 

On Sun, Mar 29, 2015 at 6:56 AM, François REMY <francois.remy.dev@outlook.com <mailto:francois.remy.dev@outlook.com> > wrote:

Hi Rick (and all the others, too ^_^),

Sorry for taking a week to reply, I really needed that time to put my thoughts in order and write them down :D 

 

I was very glad to see some work being done in order to define the Input Device drivers of the web, and I wanted to share my vision with you on the matter. Like you, I believe that Pointer Events require some more metadata (coarsity, drag inertia, …) on the devices generating the events to be able to handle new types of inputs properly and I'm a huge fan of your proposal. My free time is overbooked but if some working group discussed this further I would be happy to participate in refining your proposal.

 

Yet, I believe there's some important challenge it doesn't tackle. Something I don't like in general about Input Devices at this time is how "customized" any good device handling has to be, and how native apps often have to rely on device-specific userland drivers to harness the true device capabilities (those are not available to the web). I argue the reason of this issue is the lack of a sufficiently generic class of devices which could encompass multiple sensors and ways to interpret them, providing each app with the pieces of info it want with the granularity it needs; to sum up, I think we lack a metamodel unifying even more input devices than ever before.

 

Let's be honest, I think the proposal I'm going to make is probably far away from how most input devices are handled at the OS level at the time I write it. As a result, I'm conscient an initial implementation of the proposal will have to mock the features using the native input events of the existing platforms (a clear con). I would argue (though) that it is the case of currently implemented APIs like the Gamepad API and the PointerLock specification, from which this proposal draws ideas from.

 

However, I think the API I propose is promising and has some interesting features; though I'll of course let you be your own judge on that matter ;-) I've put toegether my ideas in a draft document [1] and an API surface [2]. 


[1] https://docs.google.com/document/d/1Ubgn0uZ73DGkBnEPKJ5CfSk-YPu5MDjq_We1gy1kFEg/edit?usp=sharing
[2] https://gist.github.com/FremyCompany/a70ee6a02d54c3d9521d

 

I would be interested to hear about what you guys think about this proposal, because I can't make this happen without you! Please feel free to comment on the Google Doc, on the Gist or by email. Please make me happy: my main intent is to initiate a discussion :-)

 

Best regards,
François

 

_________________

PS:

Please note that for the purpose of this exercice, I included devices ranging from Webcams to Touchscreens (via Depth-cams and Smart pens with gyros; but *not* next gen devices like the Leap Motion or the Kinect Skeleton Tracking, as I don't have enough experience with such devices). However I worked on cutting-edge and amazing input device scenarios like using the Mac's touchpad as a multitouch input [1], a Microsoft Touch Mouse [2], or a calibrated webcam/dephtcam [3].


[1]  <http://notebooks.com/2011/08/10/touchgrind-brings-multitouch-gaming-to-the-mac/> http://notebooks.com/2011/08/10/touchgrind-brings-multitouch-gaming-to-the-mac/
[2]  <http://www.hanselman.com/blog/AbusingTheMicrosoftResearchsTouchMouseSensorAPISDKWithAConsolebasedHeatmap.aspx> http://www.hanselman.com/blog/AbusingTheMicrosoftResearchsTouchMouseSensorAPISDKWithAConsolebasedHeatmap.aspx
[3]  <https://youtu.be/YXJ9dmxsc6w> https://youtu.be/YXJ9dmxsc6w

 

 

De : Rick Byers <mailto:rbyers@chromium.org> 
Envoyé : ‎lundi‎ ‎9‎ ‎mars‎ ‎2015 ‎16‎:‎07
À : DOM WG <mailto:www-dom@w3.org> , Gary Kacmarcik (Кошмарчик) <mailto:garykac@chromium.org> , Domenic Denicola <mailto:d@domenic.me> , Mustaq Ahmed <mailto:mustaq@chromium.org> 

 

In our latest discussion of how best to identify mouse events derived from touch events, I proposed a 'sourceDevice' property <https://lists.w3.org/Archives/Public/www-dom/2015JanMar/0052.html> .  Domenic asked <https://lists.w3.org/Archives/Public/www-dom/2015JanMar/0052.html>  for a rough sketch of what such an InputDevice API might grow to become.  Here <https://docs.google.com/a/chromium.org/document/d/1WLadG2dn4vlCewOmUtUEoRsThiptC7Ox28CRmYUn8Uw/edit>  is my first attempt at such a sketch, including some detailed references to similar APIs in other platforms.  Any thoughts?

 

Note that at the moment I'm primarily interested in standardizing and implementing the 'firesTouchEvents' bit.  However if it makes more sense for coherency, I'd also support adding (and implementing in chromium) a few of the other non-controversial pieces.

 

Rick