open/transparent reviewing (was: With footnotes (was Re: Open Access to Journal of Web Semantics (JWS)))

I found the following interesting in terms of pointers to relevant literature and overview on the topic:

https://f1000research.com/articles/6-1151/v1


I haven't done a structured survey (somebody should do one), but from all I've seen and read so far, the conclusions are at best - inconclusive. 

From own experience (on running the Semantic Web journal from scratch 10 years ago to where it is now), I believe that things like submission quality, review quality, etc. are very sensitive to detailed calibrations of the review process. E.g., some of the "negative" aspects of transparency/openness mentioned in the above referenced survey do not apply as such to the Semantic Web journal because we have calibrations in place which counteract exactly these negative effects. E.g.: The opt-out for reviewers (i.e. they can choose to remain anonymous) is extremely important for review quality and also for reviewer protection in some cases. It is hardly used, but having the option alone is already important, and in some cases it is indeed used. 

Everybody can of course have a look by themselves how well the Semantic Web journal process works - all our reviewing is done in the open.

Semantic Web Journal: http://www.semantic-web-journal.net


Best Regards,

Pascal.


--
Dr. Pascal Hitzler                   Data Semantics Laboratory
NCR Distinguished Professor          Director for Data Science
Wright State University              http://dssc.cs.wright.edu           
http://www.pascal-hitzler.de                http://daselab.org

pascal@pascal-hitzler.de   http://www.semantic-web-journal.net




>  -----Original Message-----
>  From: Harry Halpin [mailto:hhalpin@ibiblio.org]
>  Sent: Saturday, August 12, 2017 11:06 AM
>  To: Ivan Herman <ivan@w3.org>; janowicz@ucsb.edu
>  Cc: Ian Horrocks <ian.horrocks@cs.ox.ac.uk>; Sarven Capadisli
>  <info@csarven.ca>; Steffen Staab <staab@uni-koblenz.de>; semantic-
>  web@w3.org
>  Subject: Re: With footnotes (was Re: Open Access to Journal of Web
>  Semantics (JWS))
>  
>  Do we have a link or reference for the study?
>  
>  On Sat, Aug 12, 2017 at 5:04 PM Krzysztof Janowicz <janowicz@ucsb.edu
>  <mailto:janowicz@ucsb.edu> > wrote:
>  
>  
>   On 08/12/2017 06:34 AM, Ivan Herman wrote:
>  
>  
>  
>     On 11 Aug 2017, at 16:01, Harry Halpin
>  <hhalpin@ibiblio.org <mailto:hhalpin@ibiblio.org> > wrote:
>  
>     Peer review is usually not public because it can be
>  embarrassing to the author.
>  
>  
>    +1. Which also means that conscientious and mindful
>  reviewer may also decide to be less outspoken, less detailed, etc, on his/her
>  review. As a consequence the quality of the review may suffer.
>  
>  
>   We were not able to see such effect in all the years we are running
>  the Semantic Web journal and also have an anonymous review op-in option
>  used by less than 20% of all reviewers. The literature also does not confirm
>  such effect.
>  
>  
>  
>  
>     While I understand some people preferred to be
>  embarrassed in public (such as on mailing lists), this may not be in everyone's
>  interest and may not lead to more or better research,. Not sure of any
>  experimental results on the effect of open reviews on submissions or quality.
>  
>  
>    One example is anecdotical, but I would probably refrain
>  from submitting to a journal with a 100% open review, nor would I accept to
>  act as a reviewer.
>  
>    Ivan
>  
>  
>  
>  
>     On Fri, Aug 11, 2017 at 12:44 PM Sarven Capadisli
>  <info@csarven.ca <mailto:info@csarven.ca> > wrote:
>  
>  
>      On 2017-08-11 10:59, Steffen Staab wrote:
>      > I have operated the preprint server for 6
>  years.
>  
>      Cool. I stand corrected. Would you mind
>  reviewing the following statements:
>  
>      * Accepted works at JWS will have a
>  paywalled copy at Elsevier,
>      reformatted to publishers styles, PDF (and/or
>  in other formats), and
>      considered to be final, canonical, and citable.
>  
>      * Accepted works at JWS will have a free PDF
>  copy made available from
>      www.websemanticsjournal.org
>  <http://www.websemanticsjournal.org/> , but this copy is not to be cited.
>  
>      * As mentioned by Ian Horrocks, "Articles on
>  the preprint server are
>      post-review, and differ from the published
>  version only w.r.t.
>      formatting." [Pending proof]
>  
>      * Authors can choose to give Elsevier
>  exclusive rights to publish and
>      sell their work (to libraries, individuals..), or
>  authors can pay the
>      article processing charge (APC) to make the
>  works accessible for free
>      from Elsevier's access point. In the case of
>  APC, there is no constraint
>      for Elsevier to omit existing charges to
>  libraries for those works.
>  
>      * www.websemanticsjournal.org
>  <http://www.websemanticsjournal.org/>  and its archive (ie. the preprint
>  server)
>      is not funded by Elsevier, but instead funded
>  by public funds.
>  
>      * Research objects (eg underlying data, tools)
>  are not accessible, ie.
>      not hosted by www.websemanticsjournal.org
>  <http://www.websemanticsjournal.org/>  or Elsevier.
>  
>      * Article contributions to JWS may only use
>  non native Web technologies
>      eg LaTeX/Word..
>  
>      * No JWS Editor to date received payment
>  from Elsevier for their role.
>  
>      * Peer-reviews are carried out by the
>  community as opposed to Elsevier.
>      Reviewers are not paid by Elsevier (or other
>  for-profit). Reviewers may
>      be publicly funded through their academic
>  institutions or labs to carry
>      out the review process.
>  
>      * Peer-reviews for accepted and rejected
>  works are not accessible by public.
>  
>  
>      Corrections and additional information is
>  most welcome.
>  
>  
>      -Sarven
>      http://csarven.ca/#i

>  
>  
>  
>  
>  
>    ----
>    Ivan Herman, W3C
>    Publishing@W3C Technical Lead
>    Home: http://www.w3.org/People/Ivan/

>    mobile: +31-641044153
>    ORCID ID: http://orcid.org/0000-0003-0782-2704

>  
>  
>  
>  
>  
>  
>   --
>   Krzysztof Janowicz
>  
>   Geography Department, University of California, Santa Barbara
>   4830 Ellison Hall, Santa Barbara, CA 93106-4060
>  
>   Email: jano@geog.ucsb.edu <mailto:jano@geog.ucsb.edu>
>   Webpage: http://geog.ucsb.edu/~jano/

>   Semantic Web Journal: http://www.semantic-web-journal.net

Received on Saturday, 12 August 2017 17:51:08 UTC