Re: repository requirements

From: Marc Abrams (abrams@vt.edu)
Date: Tue, Apr 20 1999


Message-ID: <371CC593.6C52DBE3@vt.edu>
Date: Tue, 20 Apr 1999 14:21:07 -0400
From: Marc Abrams <abrams@vt.edu>
To: Jim Pitkow <pitkow@parc.xerox.com>, "www-wca@w3.org" <www-wca@w3.org>
Subject: Re: repository requirements

Counterpoints:

All log files used by anyone writing a paper should be in the uncertified
section of the repository.  Why?  If I read someone's paper and question the
analysis done, I would like to have access to the data used!  

Certainly for papers published in the pre-certification days access to the
(uncertified) logs used is desirable.  (Or do we retroactively certify all
logs used in the past?)

Final point.  Suppose I am a researcher and I want to do a study on X.  Turns
out there is no certified log in the wca repository on x.  What do I do?  (1)
Wait until a log gets certified and then do my study, or (2) just do the study
and try to get the log certified later?  If the world does (2), and the log
doesn't wind up certified, we again have a paper in the literature for which
the original trace data is unavailable.


Jim Pitkow wrote:
> 
> At the highest level, there are some issues as to what the requirements of
> the repository need to be.  Here's my understanding of the argument for
> having a strict policy on what enters the repository.  What I don't quite
> have is the counter argument.  If you are in the latter camp, please
> step-up and provide that line of thought.
> 
> Certified Logs or Bust!
> * Results and comparisons performed across uncertified logs are dangerous
>   because a) the data &/or descriptions may be inaccurate and  b) the results
>   not necessarily repeatable.  Such logs increase the likelihood of invalid
> theories
>   and incorrect observations being promoted.
> * As a result, scientific progress will be hampered by enabling easier
> accessibility
>   of uncertified logs.
> * Therefore, the W3C and the WCA have a responsibility to provide only
> certified logs.

-- 
Marc Abrams, Associate Professor                         
  508 McBryde Hall, Dept. of Computer Science, Virginia Tech
  Blacksburg, VA 24061-0106 U.S.A.
  voice:  (540) 231-8457, fax: (540) 231-6075