Re: Web Platform Tests Pull Request Commenting

I hadn't thought clearly about this overlap, but it's a good point, Jeff.

For the dashboard it's not terribly important that results come in fast,
one a day or so should suffice. For PRs, on the other hand, it can't really
be too fast, you could imagine pushing and getting results immediately as a
way of iterating on a test.

If the infrastructure for running the whole test suites were also flexible
enough to run individual tests on-demand and with response times no worse
than we currently get on PRs, then that sounds pretty promising indeed.

There is one thing that a PR tool would do that the dashboard does not, and
that is to compare to the results before the changes.

On Wed, May 17, 2017 at 2:10 PM Bob Holt <bob@bocoup.com> wrote:

> I was hoping someone else would get to this before me...
>
> I don't have a strong preference in terms of working on this in a silo
> versus pitching in on the dashboard.
>
> I will say that it looks like the WPT Dashboard is a view into a complete
> daily run of all web platform tests. The tool described here deals with
> aggregating the results of discrete (perhaps multiple) runs of specific
> tests tied to PRs on w3c/web-platform-tests, and commenting on those PRs. I
> think the question comes down to whether those scopes appear to overlap or
> not. I have thoughts which you can probably suss out given how I framed the
> question, but I don't think it's my call to make.
>
> If these do end up separate applications, I can imagine sharing UI
> components between the two, if nothing else but to give a consistent
> experience.
>
> On Tue, May 16, 2017 at 2:19 PM, Jeff Carpenter <jeffcarp@google.com>
> wrote:
>
>> I certainly don't want to step on any toes here, but I wanted to throw my
>> opinion out there since I see possible opportunities for saving time and
>> effort. From my perspective it looks like the project I'm working on, WPT
>> Dashboard, and this project have very similar aims:
>>
>> - WPT Results Consolidator: Collect and display WPT results
>> - WPT Dashboard: Run, collect and display WPT results
>>
>> I can't help but wonder if it would save time to build this on the same
>> infrastructure. Adding a couple of endpoints to collect and display WPT
>> results by revision would be a fairly simple addition to the dashboard. The
>> proposals, as planned, would mean we'd both be building a UI to display WPT
>> test results - this sounds like the biggest duplication of efforts.
>> Additionally the dashboard, being an App Engine app, also has the benefit
>> of not needing server setup or maintenance, or the potential to run out of
>> disk space. (see design doc here: bit.ly/wptdashboard-design-doc)
>>
>> On the other hand I completely understand there are reasons why people
>> might want these to be separate. I would love to hear your thoughts on
>> this. If this sounds like something worth pursuing further, I'm happy to
>> elaborate more specifically on how this would work technically.
>>
>> On Tue, May 9, 2017 at 6:33 AM Philip Jägenstedt <foolip@google.com>
>> wrote:
>>
>>> Very much looking forward to fewer notifications in my inbox, thanks for
>>> putting this together, Bob!
>>>
>>>
>>> On Tue, May 9, 2017 at 1:49 PM Bob Holt <bob@bocoup.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> There has been some discussion of late about the amount of commenting
>>>> done on pull requests in the w3c/web-platform-tests repository. While
>>>> various people find various comments useful, the sum of commenting can be
>>>> distracting for owners and contributors.
>>>>
>>>> After some brief discussion with James Graham and others, we're
>>>> proposing a web application that will be responsible for consolidating all
>>>> of the information currently appearing as bot comments and boiling that
>>>> information down to a single comment that it updates as results change.
>>>>
>>>> I have started a short specification document here:
>>>> https://docs.google.com/document/d/1HK5roexqU2Nd2iTMKO4UM5Gc3b5vuFm06J3E06iGcxM/edit?usp=sharing
>>>>
>>>> It should be open for public comment via that link, I am in the process
>>>> of scaffolding out the application now. I will update this document
>>>> periodically as I receive comment and make progress in the scaffolding,
>>>>
>>>> Thanks,
>>>> Bob
>>>>
>>>
>