W3C home > Mailing lists > Public > semantic-web@w3.org > August 2017

RE: open/transparent reviewing (was: With footnotes (was Re: Open Access to Journal of Web Semantics (JWS)))

From: Hitzler, Pascal <pascal.hitzler@wright.edu>
Date: Sun, 13 Aug 2017 13:00:39 +0000
To: Aldo Gangemi <aldo.gangemi@gmail.com>
CC: Harry Halpin <hhalpin@ibiblio.org>, Ivan Herman <ivan@w3.org>, "Krzysztof Janowicz" <janowicz@ucsb.edu>, Ian Horrocks <ian.horrocks@cs.ox.ac.uk>, "Sarven Capadisli" <info@csarven.ca>, Steffen Staab <staab@uni-koblenz.de>, "semantic-web@w3.org" <semantic-web@w3.org>, "alexgarciac@gmail.com" <alexgarciac@gmail.com>
Message-ID: <BN3PR01MB12864ED54B21A74F3C7C8789E18F0@BN3PR01MB1286.prod.exchangelabs.com>
>The expectations might be that “fame rewarding” and gender
>  effects could be mitigated by the potential counter-effects of exposing names
>  and/or reviews to the public, and the inherent (basically unsolvable)
>  problems of double blindness are avoided.
>  
>  Aldo

Yes.

Another aspect rarely mentioned: In a transparent setting, it is not only the public reputation of the reviewer which factors in. A journal which does transparent reviewing but does not manage to run a fair process and to make fair decisions would be easily exposable as such (all the data is public), and would thus damage its reputation. 

This does not only mean the making of decisions based on subpar reviews. It also means problems with reviewer selection, applying uniform quality thresholds for acceptance across submissions, caving in under pressure by (possibly prominent) authors or reviewers, navigating conflicts, etc. - all the stuff you see when you're PC Chair of a big conference (or EiC of a prominent journal). 

The journal furthermore publicly exposes review times, workload of each editorial board member, and gives public credit to reviewers (The Semantic Web journal even lists their names on accepted papers). So there are many more potential benefits than just looking at review quality.

Pascal.

>  -----Original Message-----
>  From: Aldo Gangemi [mailto:aldo.gangemi@gmail.com]
>  Sent: Sunday, August 13, 2017 4:41 AM
>  To: Hitzler, Pascal <pascal.hitzler@wright.edu>
>  Cc: Aldo Gangemi <aldo.gangemi@gmail.com>; Harry Halpin
>  <hhalpin@ibiblio.org>; Ivan Herman <ivan@w3.org>; Krzysztof Janowicz
>  <janowicz@ucsb.edu>; Ian Horrocks <ian.horrocks@cs.ox.ac.uk>; Sarven
>  Capadisli <info@csarven.ca>; Steffen Staab <staab@uni-koblenz.de>;
>  semantic-web@w3.org; alexgarciac@gmail.com
>  Subject: Re: open/transparent reviewing (was: With footnotes (was Re: Open
>  Access to Journal of Web Semantics (JWS)))
>  
>  One further addition/correction: I have found on arXiv [3] the description of
>  the recent controlled experiment at WSDM, and the results are that double
>  blind is less prone than single blind (not open reviewing as I wrongly reported
>  in the previous email) to reward famous authors and top institutions:
>  "Œestimated odds multipliers are 1.66 for famous authors and 1.61 and 2.10
>  for top universities and companies respectively". The authors also find a
>  possible (negative) gender effect once a meta-analysis is performed. Finally,
>  they report about the difficulties of open reviewing (content anonymization
>  and conflict declaration), which are the reasons why I was referring to the
>  “artificiality” of double blind systems.
>  
>  However, I have found no study yet comparing double blind vs. open
>  reviewing. The expectations might be that “fame rewarding” and gender
>  effects could be mitigated by the potential counter-effects of exposing names
>  and/or reviews to the public, and the inherent (basically unsolvable)
>  problems of double blindness are avoided.
>  
>  Aldo
>  
>  [3] https://arxiv.org/pdf/1702.00502.pdf

>  
>  > On 12 Aug 2017, at 20:56, Aldo Gangemi <aldo.gangemi@gmail.com>
>  wrote:
>  >
>  > I add two studies [1][2] that I have recently stumbled upon. The first one is
>  an in vivo simulation of the impact of competition on double blind
>  procedures, and suggests that competition affects peer reviewing in a bad
>  way, research wise. The second one is an Elsevier report (yes, even
>  commercial publishers seem to care about things-a-changing :)) involving five
>  journals from different disciplines experimenting with Open Reviewing (OR).
>  The results include e.g. as assessment of reviewers that either opt in or out
>  for having their names disclosed:
>  >
>  > 	• 95 percent said publishing review reports didn’t influence their
>  recommendation
>  > 	• 76 percent said the fact their reports will be publicly available
>  didn’t change their wording
>  > 	• 45 percent gave us consent to reveal their names
>  > 	• 36 percent of those who preferred to stay anonymous said they will
>  reveal their names next time they review for the journal
>  > 	• 98 percent said they will accept further review invites for the
>  > journal
>  >
>  > I also anticipate that the next ESWC2018 conference will feature Open
>  Reviewing in the three main tracks. Since OR in conferences is still rarely
>  applied, we will carefully follow what will happen, and report to the
>  community the results of the experience, and lessons learnt. For example, is
>  OR more prone to reward labs with a string reputation/influence (as Roberto
>  Navigli reported as happening in a recent WSDM conference, I will look for a
>  reference)? is OR giving us more/less detailed reviews, more/less fair or polite
>  criticism?, etc. My expectation is that, given the heterogeneity and
>  competition level within the SW community, the effects of OR will be mostly
>  positive, while avoiding the artificiality of double blind systems.
>  >
>  > Best
>  > Aldo
>  >
>  > [1]
>  > https://newrepublic.com/article/135921/science-suffering-peer-reviews-

>  > big-problems [2]
>  > https://www.elsevier.com/reviewers-update/story/innovation-in-publishi

>  > ng/is-open-peer-review-the-way-forward
>  >
>  >> On 12 Aug 2017, at 19:50, Hitzler, Pascal <pascal.hitzler@wright.edu>
>  wrote:
>  >>
>  >> I found the following interesting in terms of pointers to relevant literature
>  and overview on the topic:
>  >>
>  >> https://f1000research.com/articles/6-1151/v1

>  >>
>  >> I haven't done a structured survey (somebody should do one), but from all
>  I've seen and read so far, the conclusions are at best - inconclusive.
>  >>
>  >> From own experience (on running the Semantic Web journal from scratch
>  10 years ago to where it is now), I believe that things like submission quality,
>  review quality, etc. are very sensitive to detailed calibrations of the review
>  process. E.g., some of the "negative" aspects of transparency/openness
>  mentioned in the above referenced survey do not apply as such to the
>  Semantic Web journal because we have calibrations in place which
>  counteract exactly these negative effects. E.g.: The opt-out for reviewers (i.e.
>  they can choose to remain anonymous) is extremely important for review
>  quality and also for reviewer protection in some cases. It is hardly used, but
>  having the option alone is already important, and in some cases it is indeed
>  used.
>  >>
>  >> Everybody can of course have a look by themselves how well the Semantic
>  Web journal process works - all our reviewing is done in the open.
>  >>
>  >> Semantic Web Journal: http://www.semantic-web-journal.net

>  >>
>  >> Best Regards,
>  >>
>  >> Pascal.
>  >>
>  >>
>  >> --
>  >> Dr. Pascal Hitzler                   Data Semantics Laboratory
>  >> NCR Distinguished Professor          Director for Data Science
>  >> Wright State University              http://dssc.cs.wright.edu

>  >> http://www.pascal-hitzler.de                http://daselab.org

>  >> pascal@pascal-hitzler.de   http://www.semantic-web-journal.net

>  >>
>  >>
>  >>
>  >>> -----Original Message-----
>  >>> From: Harry Halpin [mailto:hhalpin@ibiblio.org]
>  >>> Sent: Saturday, August 12, 2017 11:06 AM
>  >>> To: Ivan Herman <ivan@w3.org>; janowicz@ucsb.edu
>  >>> Cc: Ian Horrocks <ian.horrocks@cs.ox.ac.uk>; Sarven Capadisli
>  >>> <info@csarven.ca>; Steffen Staab <staab@uni-koblenz.de>; semantic-
>  >>> web@w3.org
>  >>> Subject: Re: With footnotes (was Re: Open Access to Journal of Web
>  >>> Semantics (JWS))
>  >>>
>  >>> Do we have a link or reference for the study?
>  >>>
>  >>> On Sat, Aug 12, 2017 at 5:04 PM Krzysztof Janowicz
>  >>> <janowicz@ucsb.edu <mailto:janowicz@ucsb.edu> > wrote:
>  >>>
>  >>>
>  >>> 	On 08/12/2017 06:34 AM, Ivan Herman wrote:
>  >>>
>  >>>
>  >>>
>  >>> 			On 11 Aug 2017, at 16:01, Harry Halpin
>  <hhalpin@ibiblio.org
>  >>> <mailto:hhalpin@ibiblio.org> > wrote:
>  >>>
>  >>> 			Peer review is usually not public because it can be
>  embarrassing
>  >>> to the author.
>  >>>
>  >>>
>  >>> 		+1. Which also means that conscientious and mindful
>  reviewer may
>  >>> also decide to be less outspoken, less detailed, etc, on his/her
>  >>> review. As a consequence the quality of the review may suffer.
>  >>>
>  >>>
>  >>> 	We were not able to see such effect in all the years we are running
>  >>> the Semantic Web journal and also have an anonymous review op-in
>  >>> option used by less than 20% of all reviewers. The literature also
>  >>> does not confirm such effect.
>  >>>
>  >>>
>  >>>
>  >>>
>  >>> 			While I understand some people preferred to be
>  embarrassed in
>  >>> public (such as on mailing lists), this may not be in everyone's
>  >>> interest and may not lead to more or better research,. Not sure of
>  >>> any experimental results on the effect of open reviews on submissions or
>  quality.
>  >>>
>  >>>
>  >>> 		One example is anecdotical, but I would probably refrain
>  from
>  >>> submitting to a journal with a 100% open review, nor would I accept
>  >>> to act as a reviewer.
>  >>>
>  >>> 		Ivan
>  >>>
>  >>>
>  >>>
>  >>>
>  >>> 			On Fri, Aug 11, 2017 at 12:44 PM Sarven Capadisli
>  >>> <info@csarven.ca <mailto:info@csarven.ca> > wrote:
>  >>>
>  >>>
>  >>> 				On 2017-08-11 10:59, Steffen Staab wrote:
>  >>> 				> I have operated the preprint server for 6
>  years.
>  >>>
>  >>> 				Cool. I stand corrected. Would you mind
>  reviewing the following
>  >>> statements:
>  >>>
>  >>> 				* Accepted works at JWS will have a
>  paywalled copy at Elsevier,
>  >>> 				reformatted to publishers styles, PDF (and/or
>  in other formats),
>  >>> and
>  >>> 				considered to be final, canonical, and citable.
>  >>>
>  >>> 				* Accepted works at JWS will have a free PDF
>  copy made available
>  >>> from
>  >>> 				www.websemanticsjournal.org
>  >>> <http://www.websemanticsjournal.org/> , but this copy is not to be
>  cited.
>  >>>
>  >>> 				* As mentioned by Ian Horrocks, "Articles on
>  the preprint server
>  >>> are
>  >>> 				post-review, and differ from the published
>  version only w.r.t.
>  >>> 				formatting." [Pending proof]
>  >>>
>  >>> 				* Authors can choose to give Elsevier
>  exclusive rights to
>  >>> publish and
>  >>> 				sell their work (to libraries, individuals..), or
>  authors can
>  >>> pay the
>  >>> 				article processing charge (APC) to make the
>  works accessible for
>  >>> free
>  >>> 				from Elsevier's access point. In the case of
>  APC, there is no
>  >>> constraint
>  >>> 				for Elsevier to omit existing charges to
>  libraries for those
>  >>> works.
>  >>>
>  >>> 				* www.websemanticsjournal.org
>  >>> <http://www.websemanticsjournal.org/>  and its archive (ie. the
>  >>> preprint
>  >>> server)
>  >>> 				is not funded by Elsevier, but instead funded
>  by public funds.
>  >>>
>  >>> 				* Research objects (eg underlying data, tools)
>  are not
>  >>> accessible, ie.
>  >>> 				not hosted by www.websemanticsjournal.org
>  >>> <http://www.websemanticsjournal.org/>  or Elsevier.
>  >>>
>  >>> 				* Article contributions to JWS may only use
>  non native Web
>  >>> technologies
>  >>> 				eg LaTeX/Word..
>  >>>
>  >>> 				* No JWS Editor to date received payment
>  from Elsevier for their
>  >>> role.
>  >>>
>  >>> 				* Peer-reviews are carried out by the
>  community as opposed to
>  >>> Elsevier.
>  >>> 				Reviewers are not paid by Elsevier (or other
>  for-profit).
>  >>> Reviewers may
>  >>> 				be publicly funded through their academic
>  institutions or labs
>  >>> to carry
>  >>> 				out the review process.
>  >>>
>  >>> 				* Peer-reviews for accepted and rejected
>  works are not
>  >>> accessible by public.
>  >>>
>  >>>
>  >>> 				Corrections and additional information is
>  most welcome.
>  >>>
>  >>>
>  >>> 				-Sarven
>  >>> 				http://csarven.ca/#i

>  >>>
>  >>>
>  >>>
>  >>>
>  >>>
>  >>> 		----
>  >>> 		Ivan Herman, W3C
>  >>> 		Publishing@W3C Technical Lead
>  >>> 		Home: http://www.w3.org/People/Ivan/

>  >>> 		mobile: +31-641044153
>  >>> 		ORCID ID: http://orcid.org/0000-0003-0782-2704

>  >>>
>  >>>
>  >>>
>  >>>
>  >>>
>  >>>
>  >>> 	--
>  >>> 	Krzysztof Janowicz
>  >>>
>  >>> 	Geography Department, University of California, Santa Barbara
>  >>> 	4830 Ellison Hall, Santa Barbara, CA 93106-4060
>  >>>
>  >>> 	Email: jano@geog.ucsb.edu <mailto:jano@geog.ucsb.edu>
>  >>> 	Webpage: http://geog.ucsb.edu/~jano/

>  >>> 	Semantic Web Journal: http://www.semantic-web-journal.net

>  >>
>  >

Received on Sunday, 13 August 2017 13:01:06 UTC

This archive was generated by hypermail 2.3.1 : Sunday, 13 August 2017 13:01:16 UTC