Re: Submitting test suite results to tests.xproc.org

On that mindset ... I agree 100%
>From a standards perspective maybe performance should be left out completely 
and left up to the vendors to publish themselves.  There is too much "gray 
area" in performance results that unless you spec out extremely well exactly 
what it all means, not only may it discourage publication, but could easily 
be abused either intentionally or unintentionally.

Compliance with specs via comparison of input/vs output is fairly objective 
(although some things are hard like validating that a remote push/write 
request actually happened) but validating performance results is neigh 
impossible.    I agree this should be outside the scope of the specs and W3C 
...

( Unless someone wants to write a NEW spec on how to actually perform 
benchmarks and validate them !)




------------ quoteth
"David A. Lee" <dlee@calldei.com> writes:
> Alternatively ... maybe this is more valuable as a seperate report
> entirely with performance benchmarks as the goal not spec compliance.

I think that would be better as a separate report. With my chair's hat
on, I don't want to ask implementors to report performance numbers
because I fear it will make implementors reluctant to submit their
results.

But I don't mind setting up a second report when I've got the time.
And some more appopriate tests.

Received on Friday, 5 December 2008 19:33:28 UTC