Re: [DOM3Events] InputDevice API sketch

On Thu, Apr 9, 2015 at 8:44 AM, François REMY <francois.remy.dev@outlook.com
> wrote:

>  Wow, thank you very much for this well-balanced reply. I really
> appreciate the time you must have taken to write it, not even counting the
> time you probably took to have a look at the proposal in itself. ❤
>
> To be completely honest, I somehow knew this wasn’t the right place to
> make this proposal because, to get there, the very first step is to get a
> new Input Device Driver standards which allows for (at least some of) such
> features to be exposed at the OS level. The thing is: I don’t know anyone
> from that field, and even if some people there agreed to work on it, it’s
> definitely going to take years to get to a point where the design I propose
> could give us any yield. I recognize that.
>
> The real purpose of my intervention was mostly to expose real use-cases
> I’ve seen working (like Touch Input on a trackpad, on a Touch Mouse, via a
> Depth Camera, what happens when you have two mices connected, etc) and make
> sure we have them in mind when making additional incremental changes to the
> platform.
>
> *In short: I want to make sure we adopt when it’s possible a cross-device
> vision and not a per-device vision* (like most of the API have at this
> point).
>
> An example of this philosophy would be to expose properties like
> “inputDevice.defaultLongPress”, “defaultLongPress.timeout”, “defaultLongPress.maxIgnoredMoveDistance”
> etc which allows a PointerEvent listener to have the right “tap&hold”
> feeling for the Device+OS combination (if any is necessary) without having
> to test “is this device a touch device” or “is this device a mouse?”. Ditto
> for scroll/manipulation APIs. We want to know if a device is expected to
> trigger panning and (if yes) with which initial move threshold tolerance.
> Currently we only have the choice between a mouse which doesn’t need any of
> this and a touch screen which does. A new input device will have to comply
> with either one of those contracts. #SadPanda
>
> I hate very much the idea to have an event handler “customized” for each
> pointing device type, and not relying on device properties (and right now
> this is how a PointerEvent-consuming code looks like). That makes it hard
> to introduce new types of input devices. Let’s try avoid any new “Touch
> Input fires ontouchmove then onmousemove” fiasco with new input devices in
> the future.
>
> *To sum up, I would like to take advantage of the fact you propose to
> define InputDevices to, at the very least, expose behaviors & behavioral
> properties, and not device types* (when it’s reasonably possible).
>

> This requires some more work, but I think this work is necessary to open
> doors to the future. Let’s not be too minimalist when we want to be
> realistic ;-)
>

Yes, I agree this is a good direction in general - have developers ask the
thing they really want to know (does dragging this pointer cause
scrolling), rather than infer it from some related property (is this
pointer a touch).  I actually raised concerns about
PointerEvent.pointerType during the original standardization (i.e. that no
new device would really end up choosing a value other than the original 3
because it would break some subset of websites).  But we agreed we probably
wanted to keep it for close compat with IE 10, and we could add the more
generic properties I suggested as future devices became standardized.  This
is why we have 'pointer' and 'hover' media queries instead of
'hasTouchscreen' and 'hasMouse'.  I consider this InputDevice API to be the
next stage of that debate.

Each possible API will need independent debate though.  Eg. I personally
don't think a "move threshold tolerance" API is a good idea - I believe
Safari, for example, relies on an algorithm that's more complex than a
simple bounding box.  For that particular use case we've instead discussed
an API like TouchEvent.willTriggerScroll - so that for a given event you
can see if it's about to cause a scroll to occur.  I'm not sure offhand how
to map that idea to pointer events (which never block scrolling), but given
a suitable concrete and important use case I'm confident we can.

That said, back compat is always going to be an issue that's impossible to
avoid completely.  New interaction modes may need to differentiate between
behavior needed for compatibility and the true behavior intended for code
that is aware of the device type. We can do our best to minimize this with
some upfront thought, but future-proofing is impossible and guessing too
hard can add unnecessary complexity and confusion without ever actually
achieving the goal of being entirely future-proof.

Let's not forget that what makes the web truly unique is the incredible
reach, interoperability and lack of centralized control.  Dealing
pragmatically with decades-long back-compat challenges is the (large) price
we pay for that benefit.  Sure it would be nice to periodically deprecate
all the old cruft and replace it with newly designed API surface (like
native platforms tend to do in some way every ~5-10 years).  But to me that
annoying cost is worth the huge benefit.  As you know, taking even the
small leap here needed for Pointer Events is still far from being a
platform success story, but we're trying ;-)


> François
>
>
> PS: More comments inline tagged with [REPLY] but those are less important
>
>
> *De :* Rick Byers <rbyers@chromium.org>
> *Envoyé :* ‎jeudi‎ ‎9‎ ‎avril‎ ‎2015 ‎04‎:‎16
> *À :* François REMY <francois.remy.dev@outlook.com>
> *Cc :* DOM WG <www-dom@w3.org>, public-pointer-events@w3.org, Gary
> Kacmarcik (Кошмарчик) <garykac@chromium.org>, Domenic Denicola
> <d@domenic.me>, Mustaq Ahmed <mustaq@chromium.org>
>
> Thanks for your detailed response François, and sorry it took me so long
> to reply.
>
> I like the idea of provide more abstract input device information, and
> there are a number of nice properties in your design.  However it's hard
> for me to imagine a big highly-abstract API like this really being
> successfully deployed to the web platform.  It's not enough for a browser
> to just implement this API (as you say, mostly hard-coding it based on the
> less-abstract APIs provided by the host OS), but to be really valuable each
> browser would have to also do work per input device.  In practice I'm not
> sure we'd really get browser vendors excited enough to implement more than
> the most common devices.
>
> [REPLY] I shall ship beer and cool input devices to all browsers vendors
> daily until either excitement arises or my money stock vanishes ;-)
>
> I'm personally most interested in incremental approaches where we can add
> some small surface area but have the ability to efficiently iterate adding
> small feature after feature as the need arises (without being afraid to
> even approach the standards group due to the expected effort required).
> I.e. after months of debate, let's see if we can get even my simple "does
> this device fire touch events" issue addressed by a W3C spec before we
> attempt something with orders of magnitude more abstraction and
> bikeshedding opportunities ;-).  I have low confidence that we could design
> an API in advance which will automatically work great for new devices
> without API changes (eg. even your highly generalized API is probably no
> better at supporting the new Mac force touch touchpad than pointer events
> would be).  So to me being able to iterate efficiently is more important
> that predicting all possible use cases in advance with a highly abstract
> API.
>
> [REPLY] Yes, definitely.
>
> However, I think this just highlights the problem of the specialist
> scenarios being blocked on the standards consensus process and browser
> prioritization funnel (which - when it works at all - really works only for
> the most common use cases).
>
> [REPLY] +1
>
> As long as the browsers need to add code to support each new type of input
> device (especially when new API surface is involved), we're always going to
> be playing catch-up (eg. imagine if leap motion couldn't successfully ship
> a device without convincing the Windows, Darin and Linux kernel teams to
> implement some feature, blocking on adequate deployment of updated OS
> versions).  Personally I believe the only practical path to the sort of
> fully generic device support utopia you're advocating for here is via
> simpler standard primitives.  Eg. perhaps with low-level web bluetooth and
> USB APIs you could implement the abstract API you're looking for here in a
> JS library?  Once a particular device/scenario gains popularity we can then
> justify investments to make that particular scenario better (eg. perhaps
> with a higher level API that can be implemented without bothering the user
> with a permissions prompt).
>
> [REPLY] This means more special-cased code, though. At the very least, we
> should provide a way to fire consistent polyfilled Pointer/Touch events.
> What about PointerEvent.registerCustomPointer() which would return a free
> pointer id, and unregisterCustomPointer(id) which would free it?
>
> Still I think there are other ways to make pragmatic incremental progress
> towards some of your goals.
>
> [REPLY] I hope so
>
> Eg. we've talked about raw touchpad access in PEWG a few times, and it's
> something I'm moderately interested in seeing happen (eg. for handwriting
> recognition scenarios).  I believe Jacob (PE editor) and I agree that a
> small incremental extension to PointerEvents / pointer lock could address
> that use case - but we'd really need a champion for the use case to drive
> it in the WG (Microsoft doesn't generally have adequate touchpad APIs /
> hardware, Google cares most about phones, and Apple doesn't participate in
> PE).  If anyone is interested in helping push this (especially someone that
> would actually consume the API in a shipping product),
> public-pointer-events is the place to do so (I promise to help facilitate).
>
> [REPLY] That won’t be me, but it’s great to see people are open to it.
>
> So I think it's most important that we establish some minimally abstract
> framework that is simple while still flexible enough to cover the variety
> of most likely use cases.  I think PointerEvents is part of this, but we
> always knew we were missing some place to hang device query APIs off.
> Hopefully a simple InputDevice API can give us the other half.  This still
> doesn't do much to help non-pointer devices but perhaps that's not critical
> enough to worry too much about - I don't know.
>
> [REPLY] At least, this will make things better for what exists today, so
> it's definitely a win. Other input devices still have work to do before we
> can consider them standards, so it’s fine delaying.
>
> Sorry not to be more positive - just trying to be realistic about how the
> web evolves in practice...
>
> [REPLY] The contrary would have surprised me, to be honest. I used
> the “make people see your idea's big picture, to make it easier for them to
> understand it while micro-applied to smaller cases” strategy here.
>
> [REPLY] Thank you very much again for allocating some of your time to this.
>
> Rick
>
>
>
>
>
> On Sun, Mar 29, 2015 at 6:56 AM, François REMY <
> francois.remy.dev@outlook.com> wrote:
>
>>  Hi Rick (and all the others, too ^_^),
>> *Sorry for taking a week to reply, I really needed that time to put my
>> thoughts in order and write them down :D *
>>
>> *I was very glad to see some work being done in order to define the Input
>> Device drivers of the web*, and I wanted to share my vision with you on
>> the matter. Like you, I believe that Pointer Events require some more
>> metadata (coarsity, drag inertia, …) on the devices generating
>> the events to be able to handle new types of inputs properly and I'm a huge
>> fan of your proposal. My free time is overbooked but if some working group
>> discussed this further I would be happy to participate in refining your
>> proposal.
>>
>> *Yet, I believe there's some important challenge it doesn't tackle*.
>> Something I don't like in general about Input Devices at this time is how
>> "customized" any good device handling has to be, and how native apps often
>> have to rely on device-specific userland drivers to harness the true device
>> capabilities (those are not available to the web). I argue the reason of
>> this issue is the lack of a sufficiently generic class of devices which
>> could encompass multiple sensors and ways to interpret them, providing each
>> app with the pieces of info it want with the granularity it needs; to sum
>> up, I think we lack a metamodel unifying even more input devices than ever
>> before.
>>
>> *Let's be honest*, I think the proposal I'm going to make is
>> probably far away from how most input devices are handled at the OS level
>> at the time I write it. As a result, I'm conscient an initial
>> implementation of the proposal will have to mock the features using the
>> native input events of the existing platforms (a clear con). I would argue
>> (though) that it is the case of currently implemented APIs like the Gamepad
>> API and the PointerLock specification, from which this proposal draws ideas
>> from.
>>
>> *However, I think the API I propose is promising *and has some
>> interesting features; though I'll of course let you be your own judge on
>> that matter ;-) I've put toegether my ideas in a draft document [1] and an
>> API surface [2].
>>
>> [1]
>> https://docs.google.com/document/d/1Ubgn0uZ73DGkBnEPKJ5CfSk-YPu5MDjq_We1gy1kFEg/edit?usp=sharing
>> [2] https://gist.github.com/FremyCompany/a70ee6a02d54c3d9521d
>>
>> *I would be interested to hear about what you guys think *about this
>> proposal, because I can't make this happen without you! Please feel free to
>> comment on the Google Doc, on the Gist or by email. Please make me happy:
>> my main intent is to initiate a discussion :-)
>>
>> Best regards,
>> François
>>
>> _________________
>> *PS:*
>> Please note that for the purpose of this exercice, I included devices
>> ranging from Webcams to Touchscreens (via Depth-cams and Smart pens with
>> gyros; but *not* next gen devices like the Leap Motion or the Kinect
>> Skeleton Tracking, as I don't have enough experience with such devices).
>> However I worked on cutting-edge and amazing input device scenarios like
>> using the Mac's touchpad as a multitouch input [1], a Microsoft Touch Mouse
>> [2], or a calibrated webcam/dephtcam [3].
>>
>> [1]
>> *http://notebooks.com/2011/08/10/touchgrind-brings-multitouch-gaming-to-the-mac/*
>> <http://notebooks.com/2011/08/10/touchgrind-brings-multitouch-gaming-to-the-mac/>
>> [2]
>> *http://www.hanselman.com/blog/AbusingTheMicrosoftResearchsTouchMouseSensorAPISDKWithAConsolebasedHeatmap.aspx*
>> <http://www.hanselman.com/blog/AbusingTheMicrosoftResearchsTouchMouseSensorAPISDKWithAConsolebasedHeatmap.aspx>
>> [3] *https://youtu.be/YXJ9dmxsc6w* <https://youtu.be/YXJ9dmxsc6w>
>>
>>
>> *De :* Rick Byers <rbyers@chromium.org>
>> *Envoyé :* ‎lundi‎ ‎9‎ ‎mars‎ ‎2015 ‎16‎:‎07
>> *À :* DOM WG <www-dom@w3.org>, Gary Kacmarcik (Кошмарчик)
>> <garykac@chromium.org>, Domenic Denicola <d@domenic.me>, Mustaq Ahmed
>> <mustaq@chromium.org>
>>
>> In our latest discussion of how best to identify mouse events derived
>> from touch events, I proposed a 'sourceDevice' property
>> <https://lists.w3.org/Archives/Public/www-dom/2015JanMar/0052.html>.  Domenic
>> asked <https://lists.w3.org/Archives/Public/www-dom/2015JanMar/0052.html>
>> for a rough sketch of what such an InputDevice API might grow to become.
>> Here
>> <https://docs.google.com/a/chromium.org/document/d/1WLadG2dn4vlCewOmUtUEoRsThiptC7Ox28CRmYUn8Uw/edit#>
>> is my first attempt at such a sketch, including some detailed references to
>> similar APIs in other platforms.  Any thoughts?
>>
>> Note that at the moment I'm primarily interested in standardizing and
>> implementing the 'firesTouchEvents' bit.  However if it makes more sense
>> for coherency, I'd also support adding (and implementing in chromium) a few
>> of the other non-controversial pieces.
>>
>> Rick
>>
>>
>