Re: [w3c/uievents] Expose 'inertial scrolling state' in wheel events (#58)

It's nice to know I'm not the only one that would like this. I don't encourage "scroll hijacking" but I think it's still useful to know if a wheel event was user initiated vs automatic. Especially for in-browser game controllers, but it would also make carousel components more robust by allowing us to compute final resting positions based on when the user stopped touching their device vs waiting for the inertial scrolling to finish and then recentering the carousel afterwards which always looks a bit shabby.

In any case, I don't think Apple or W3C will begin exposing this data anytime soon so I've begun work on a way around the issue and am looking for others to help collaborate on it. I'm using tensorflow.js to train a model on sequences of delta values and time offsets so that I can present it with new sequences and let it predict if they are real (user initiated) or fake (initiated by the operating system). 

My approach is quite simple: I use a frontend website with a <div> that accepts wheel events and ask the user to hold down the shift key while they are physically touching their trackpad. As they scroll, they keep the shift key held down and when they release (and the synthetic events begin) they release the shift key. This leaves us with computable sequences of `wheel delta` and `time offset` values that we know to be real vs fake. 

So far it's showing promising results, however there are some external factors that still need to be mediated. Most notably, the prediction itself should be moved to a separate thread using a webworker so that it does not interfere with the sequence time offset values. I've observed that the automatic wheel events will issue higher delta values if it sees that it's been blocked for longer than a normal animation frame duration. Another big help I could use would be on the training end as I'm using mostly default values for my AI model layers and someone out there might know of a more accurate setup. 

Here is a link to the public repository:
https://gitlab.com/daveseidman/realwheel-mac

And here is a screen recording of the system in action: https://drive.google.com/file/d/1AKM13MYigFjPVKCeQCUZgTMnEfez_etk

During the first half of the video I'm holding shift while "scrolling" on my trackpad, holding shift during the training portion moves the slider all the way to the right to indicate that this is a real wheel event. When I release the shift key the slider goes all the way to the left indicating these are synthetic events. After a debounced timeout, the slider returns to the middle. 

In the second half I begin training the system on the sequences it pulled from my training data which is visualized with TensorBoard. 

In the last 15 seconds I'm "scrolling" on my trackpad again, but this time I'm not pressing the shift key at all since the system is now predicting whether the input is real or synthesized. You can see the slider jumps to the right as my scroll events start and then back left when I remove my fingers. Obviously without seeing my trackpad this is difficult to visualize so I'd encourage you to download the repo and try it yourself. I may also re-record the process along with a camera feed pointed at my hands. 

There is definitely a slight delay (about 500ms) between when I stop touching the trackpad and when the slider starts moving to the left, but I believe adjusting where I set the "windows" on my training data could improve that. I also think training with much more data would help as well. 

-- 
Reply to this email directly or view it on GitHub:
https://github.com/w3c/uievents/issues/58#issuecomment-1791314554
You are receiving this because you are subscribed to this thread.

Message ID: <w3c/uievents/issues/58/1791314554@github.com>