Re: Ill-defined programs

On Mon, Nov 13, 2017 at 7:05 PM, Maciej Stachowiak <mjs@apple.com> wrote:

>
>
> On Nov 13, 2017, at 6:50 PM, Corentin Wallez <cwallez@google.com> wrote:
>
> Hey Maciej,
>
> WebGL already has parts that follow (2.5) when accesses to arbitrary
> offsets are possible. In WebGL 2 this is only limited to vertex buffers and
> arrays in uniform buffers. In WebGPU with compute functionality, accesses
> to arbitrary offset become more common with the "shader storage buffers"
> and "texture read-write" features. When possible we would like to use
> hardware features to secure these accesses as this will get us the most
> performance. This is where (2.5) comes from because different HW implement
> the "robust access" feature differently.
>
> This interop problem hasn't been a problem because shaders that hits any
> of the cases of (2.5) will usually produce very incorrect and noticeable
> results.
>
>
> Will this still be true when more kinds of shaders can hit (2.5)?
>
> The only data I have about this is that most of the portability issues
native developers are talking about are centered on the API, not so much on
the shaders. So not clear.


>  - Maciej
>
>
>
> Corentin
>
> On Mon, Nov 13, 2017 at 6:03 PM, Maciej Stachowiak <mjs@apple.com> wrote:
>
>>
>> Hi Jeff,
>>
>> I think we might be making different assumptions about the amount of
>> undefinedness that is being proposed:
>>
>> (1) Approximately as interoperable as WebGL.
>>
>> (2) if a shader program reads out of bounds or violates any other
>> correctness constraint, then all bets are off.
>>
>> (2.5)  if a shader program reads out of bounds or violates any other
>> correctness constraint,, then it might do any one of a fixed list of
>> behaviors (e.g. terminate, clamp to bounds, or return constant zero).
>>
>>
>> I'm reluctantly ok with (1). I'd hope we could do better, but it might be
>> that this is as good as it gets. Real-World interop for WebGL is ok, even
>> if it's not as good as some other older web technologies.
>>
>> My impression is that what Dzhmitry was advocating was more like (2): "We
>> don't care about the computed results or performance of an ill-behaved
>> application."
>>
>> My impression is that WebGL does not follow approach (2). The behavior
>> for ill-defined applications is specified and covered by tests, and when
>> there are behavior differences, it's due to things like GPUs not doing
>> floating point arithmetic consistently. It's not due to completely
>> different behavior for blown bounds checks.
>>
>> I don't think (2) or even (2.5) is ok. It seems likely to create a much
>> bigger interop problem than we have with WebGL today. It's not ok to
>> completely abandon interop for programs with a certain class of bugs. Most
>> nontrivial programs will be buggy, and many will com to inadvertently
>> depend on their bugs.
>>
>> Regards,
>> Maciej
>>
>>
>>
>> > On Nov 13, 2017, at 4:37 PM, Jeff Gilbert <jgilbert@mozilla.com> wrote:
>> >
>> > We (Mozilla at least) definitely feel that WebGL 1 and 2 are in a
>> > great place portability-wise, and this is also the feedback we get
>> > from devs and partners we've worked with. WebGL has real
>> > implementation differences, with real portability concerns, but these
>> > have not become malignant. Interop is a very real concern, but there
>> > is middle ground between wild-wild-west (which absolutely no one here
>> > is proposing) and absolute portability for all apps. We have been very
>> > successful in striking that middle ground with WebGL, and I don't see
>> > that changing with WebGPU. Centralized test suites ensure all points
>> > of portability we require, and prevent implementations from deviating
>> > from the spec, and guiding them back into compliance.
>> >
>> > People write browser-specific code all the time, both on accident and
>> > on purpose. There's no panacea for it. We should make smart choices
>> > and properly weight the costs, as well as the benefits, of various
>> > approaches. Flatly refusing to consider slight deviations is naive,
>> > and not supported by our existing experience in this area.
>> >
>> > I'm trying to avoid rehashing years of API decisions here, and largely
>> > decisions we're happy with. I understand your concerns, but they are
>> > largely (and fortunately) not reflected in our experience with WebGL,
>> > which is the closest thing to prior art that we have here.
>> >
>> > I do not think we disagree as much as you fear, but I simply cannot
>> > agree with how hard you are pushing on excruciating portability. There
>> > is a balance to strike, and we've been fairly successful at doing so.
>> >
>> > On Mon, Nov 13, 2017 at 4:02 PM, Maciej Stachowiak <mjs@apple.com>
>> wrote:
>> >>
>> >>
>> >>> On Nov 13, 2017, at 3:44 PM, Jeff Gilbert <jgilbert@mozilla.com>
>> wrote:
>> >>>
>> >>> First off, those using these APIs (WebGL and WebGPU) are not writing
>> >>> geocities websites. They are engineers, and we should optimize for
>> >>> these engineers, not for the layman.
>> >>
>> >> The worst interop problems come from extremely popular sites created
>> by professional engineers, not geocities sites made by laymen.
>> >>
>> >>>
>> >>> In WebGL, we are aggressive about issuing descriptive errors and
>> >>> warnings on malformed or questionable API use. We've found this
>> >>> quickly surfaces portability issues, and makes them very quick to
>> >>> solve. I anticipate continuing this behavior during implementation of
>> >>> WebGPU. We have a ton of tools for giving feedback to devs who are
>> >>> doing the wrong thing, particularly if we have a less-perf-sensitive
>> >>> debug mode. (which can even be a JS library, not needing to be baked
>> >>> into the browser)
>> >>>
>> >>> I think good validation is important for development, but its
>> >>> performance impact can hurt outside development, once a product is
>> >>> released. In our usage of APIs inside the browsers, outside of DEBUG
>> >>> builds, we try to turn off as much validation code as we can, given
>> >>> its performance impact. If we find it useful to do this, why would web
>> >>> apps not find it useful to do similar, and be able to run with a
>> >>> minimum (but not zero!) level of validation?
>> >>
>> >> They'll definitely want to do this. It's just a bad idea to let them,
>> because they will accidentally code sites that only work in one browser, or
>> only on one OS, or only one CPU architecture, or only one GPU. Some of this
>> may be inevitable but we should try our hardest to limit it.
>> >>
>> >>>
>> >>> In the six years I've been a part of WebGL, we have not seen the
>> >>> divergence in behaviors that you are afraid of, so I'm not sure why
>> >>> you think they will happen with this new API. Please appreciate our
>> >>> experience with this field, and the differences it has as compared to
>> >>> other areas.
>> >>
>> >> According to Ken, it seems like WebGL has tried a lot harder to
>> guarantee interop than the standard suggested here.
>> >>
>> >> Even so, we have seen WebGL sites that only work in Chrome. So WebGL
>> doesn't prove that interop isn't a concern for graphics.
>> >>
>> >>
>> >>>
>> >>> On Mon, Nov 13, 2017 at 3:29 PM, Maciej Stachowiak <mjs@apple.com>
>> wrote:
>> >>>>
>> >>>>
>> >>>>> On Nov 13, 2017, at 3:21 PM, Jeff Gilbert <jgilbert@mozilla.com>
>> wrote:
>> >>>>>
>> >>>>> On Mon, Nov 13, 2017 at 12:27 PM, Maciej Stachowiak <mjs@apple.com>
>> wrote:
>> >>>>>> On Nov 13, 2017, at 11:44 AM, Dzmitry Malyshau <
>> dmalyshau@mozilla.com>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>> Yes. Strong objection. If behavior of any programs is not fully
>> specified,
>> >>>>>> then web developers will start to accidentally depend on the
>> behavior of one
>> >>>>>> browser (usually whichever is most popular), and then browsers
>> will have to
>> >>>>>> reverse-engineer each others' behavior. This has happened so many
>> times in
>> >>>>>> the course of web standards development that it's almost a running
>> joke.
>> >>>>>> Every once in a while someone says "hey, let's just not define
>> error
>> >>>>>> handling, we only need to define the behavior for valid content"
>> it happens.
>> >>>>>> The first time was HTML, Browsers ended up reverse-engineering
>> each other's
>> >>>>>> error handling until finally they got sick of the W3C not defining
>> this and
>> >>>>>> formed the WHATWG to create HTML5, which fully specified parsing
>> behavior
>> >>>>>> for all invalid documents. CSS, JavaScript and WebAssembly also
>> have fully
>> >>>>>> interoperable behavior by spec, even in "invalid" or "error" or
>> >>>>>> "ill-defined" cases.
>> >>>>>>
>> >>>>>> Let's not make this rookie mistake. We must fully define the
>> behavior of all
>> >>>>>> programs.
>> >>>>>
>> >>>>> We can fully define it without requiring exact behavior.
>> >>>>
>> >>>> I'm not sure what that means. Are you suggesting a menu choice of
>> one of N behaviors? Then the most popular behavior will become a de facto
>> standard.
>> >>>>
>> >>>>> It's not a rookie mistake to make compromises here.
>> >>>>
>> >>>> It totally is. I can't think of a time we've done this on the web
>> and it has been ok in the long run. I gave you a very notable failure
>> example. Can you cite any past successes for the strategy of sacrificing
>> interoperability on the web?
>> >>>>
>> >>>>> There are a variety of ways
>> >>>>> in which our API here, like WebGL, differs greatly from the parsing
>> >>>>> deviations in early HTML. It is not white and black. Short of proved
>> >>>>> code, there will always be accidental portability problems, even if
>> >>>>> from nothing other than cargo-culting. We should make smart
>> >>>>> compromises here, using all the tools and avenues available to us,
>> >>>>> rather than have a knee-jerk requirement for maximum portability.
>> >>>>> (absolute portability is not even possible with our base graphics
>> >>>>> APIs)
>> >>>>
>> >>>> It's my impression that, historically, graphics programmers have
>> accepted a much lower bar for portability than the standard we strive for
>> on the web. I believe we should shoot for something much closer to the web
>> bar for portability. Accidental portability problems from cargo-culting are
>> exactly the kind of problems I am worried about. The more we create the
>> opportunity for such problems, the more likely that in 5-10 years we'll all
>> have to reverse engineer Chrome for Windows and Safari for iOS behavior for
>> desktop and mobile respectively.
>> >>>>
>> >>>> Regards,
>> >>>> Maciej
>> >>>>
>> >>>
>> >>
>>
>>
>>
>
>