- From: Aldo Gangemi <aldo.gangemi@gmail.com>
- Date: Sun, 13 Aug 2017 10:40:55 +0200
- To: Pascal Hitzler <pascal.hitzler@wright.edu>
- Cc: Aldo Gangemi <aldo.gangemi@gmail.com>, Harry Halpin <hhalpin@ibiblio.org>, Ivan Herman <ivan@w3.org>, Krzysztof Janowicz <janowicz@ucsb.edu>, Ian Horrocks <ian.horrocks@cs.ox.ac.uk>, Sarven Capadisli <info@csarven.ca>, Steffen Staab <staab@uni-koblenz.de>, "semantic-web@w3.org" <semantic-web@w3.org>, alexgarciac@gmail.com
One further addition/correction: I have found on arXiv [3] the description of the recent controlled experiment at WSDM, and the results are that double blind is less prone than single blind (not open reviewing as I wrongly reported in the previous email) to reward famous authors and top institutions: "estimated odds multipliers are 1.66 for famous authors and 1.61 and 2.10 for top universities and companies respectively". The authors also find a possible (negative) gender effect once a meta-analysis is performed. Finally, they report about the difficulties of open reviewing (content anonymization and conflict declaration), which are the reasons why I was referring to the “artificiality” of double blind systems. However, I have found no study yet comparing double blind vs. open reviewing. The expectations might be that “fame rewarding” and gender effects could be mitigated by the potential counter-effects of exposing names and/or reviews to the public, and the inherent (basically unsolvable) problems of double blindness are avoided. Aldo [3] https://arxiv.org/pdf/1702.00502.pdf > On 12 Aug 2017, at 20:56, Aldo Gangemi <aldo.gangemi@gmail.com> wrote: > > I add two studies [1][2] that I have recently stumbled upon. The first one is an in vivo simulation of the impact of competition on double blind procedures, and suggests that competition affects peer reviewing in a bad way, research wise. The second one is an Elsevier report (yes, even commercial publishers seem to care about things-a-changing :)) involving five journals from different disciplines experimenting with Open Reviewing (OR). The results include e.g. as assessment of reviewers that either opt in or out for having their names disclosed: > > • 95 percent said publishing review reports didn’t influence their recommendation > • 76 percent said the fact their reports will be publicly available didn’t change their wording > • 45 percent gave us consent to reveal their names > • 36 percent of those who preferred to stay anonymous said they will reveal their names next time they review for the journal > • 98 percent said they will accept further review invites for the journal > > I also anticipate that the next ESWC2018 conference will feature Open Reviewing in the three main tracks. Since OR in conferences is still rarely applied, we will carefully follow what will happen, and report to the community the results of the experience, and lessons learnt. For example, is OR more prone to reward labs with a string reputation/influence (as Roberto Navigli reported as happening in a recent WSDM conference, I will look for a reference)? is OR giving us more/less detailed reviews, more/less fair or polite criticism?, etc. My expectation is that, given the heterogeneity and competition level within the SW community, the effects of OR will be mostly positive, while avoiding the artificiality of double blind systems. > > Best > Aldo > > [1] https://newrepublic.com/article/135921/science-suffering-peer-reviews-big-problems > [2] https://www.elsevier.com/reviewers-update/story/innovation-in-publishing/is-open-peer-review-the-way-forward > >> On 12 Aug 2017, at 19:50, Hitzler, Pascal <pascal.hitzler@wright.edu> wrote: >> >> I found the following interesting in terms of pointers to relevant literature and overview on the topic: >> >> https://f1000research.com/articles/6-1151/v1 >> >> I haven't done a structured survey (somebody should do one), but from all I've seen and read so far, the conclusions are at best - inconclusive. >> >> From own experience (on running the Semantic Web journal from scratch 10 years ago to where it is now), I believe that things like submission quality, review quality, etc. are very sensitive to detailed calibrations of the review process. E.g., some of the "negative" aspects of transparency/openness mentioned in the above referenced survey do not apply as such to the Semantic Web journal because we have calibrations in place which counteract exactly these negative effects. E.g.: The opt-out for reviewers (i.e. they can choose to remain anonymous) is extremely important for review quality and also for reviewer protection in some cases. It is hardly used, but having the option alone is already important, and in some cases it is indeed used. >> >> Everybody can of course have a look by themselves how well the Semantic Web journal process works - all our reviewing is done in the open. >> >> Semantic Web Journal: http://www.semantic-web-journal.net >> >> Best Regards, >> >> Pascal. >> >> >> -- >> Dr. Pascal Hitzler Data Semantics Laboratory >> NCR Distinguished Professor Director for Data Science >> Wright State University http://dssc.cs.wright.edu >> http://www.pascal-hitzler.de http://daselab.org >> pascal@pascal-hitzler.de http://www.semantic-web-journal.net >> >> >> >>> -----Original Message----- >>> From: Harry Halpin [mailto:hhalpin@ibiblio.org] >>> Sent: Saturday, August 12, 2017 11:06 AM >>> To: Ivan Herman <ivan@w3.org>; janowicz@ucsb.edu >>> Cc: Ian Horrocks <ian.horrocks@cs.ox.ac.uk>; Sarven Capadisli >>> <info@csarven.ca>; Steffen Staab <staab@uni-koblenz.de>; semantic- >>> web@w3.org >>> Subject: Re: With footnotes (was Re: Open Access to Journal of Web >>> Semantics (JWS)) >>> >>> Do we have a link or reference for the study? >>> >>> On Sat, Aug 12, 2017 at 5:04 PM Krzysztof Janowicz <janowicz@ucsb.edu >>> <mailto:janowicz@ucsb.edu> > wrote: >>> >>> >>> On 08/12/2017 06:34 AM, Ivan Herman wrote: >>> >>> >>> >>> On 11 Aug 2017, at 16:01, Harry Halpin >>> <hhalpin@ibiblio.org <mailto:hhalpin@ibiblio.org> > wrote: >>> >>> Peer review is usually not public because it can be >>> embarrassing to the author. >>> >>> >>> +1. Which also means that conscientious and mindful >>> reviewer may also decide to be less outspoken, less detailed, etc, on his/her >>> review. As a consequence the quality of the review may suffer. >>> >>> >>> We were not able to see such effect in all the years we are running >>> the Semantic Web journal and also have an anonymous review op-in option >>> used by less than 20% of all reviewers. The literature also does not confirm >>> such effect. >>> >>> >>> >>> >>> While I understand some people preferred to be >>> embarrassed in public (such as on mailing lists), this may not be in everyone's >>> interest and may not lead to more or better research,. Not sure of any >>> experimental results on the effect of open reviews on submissions or quality. >>> >>> >>> One example is anecdotical, but I would probably refrain >>> from submitting to a journal with a 100% open review, nor would I accept to >>> act as a reviewer. >>> >>> Ivan >>> >>> >>> >>> >>> On Fri, Aug 11, 2017 at 12:44 PM Sarven Capadisli >>> <info@csarven.ca <mailto:info@csarven.ca> > wrote: >>> >>> >>> On 2017-08-11 10:59, Steffen Staab wrote: >>> > I have operated the preprint server for 6 >>> years. >>> >>> Cool. I stand corrected. Would you mind >>> reviewing the following statements: >>> >>> * Accepted works at JWS will have a >>> paywalled copy at Elsevier, >>> reformatted to publishers styles, PDF (and/or >>> in other formats), and >>> considered to be final, canonical, and citable. >>> >>> * Accepted works at JWS will have a free PDF >>> copy made available from >>> www.websemanticsjournal.org >>> <http://www.websemanticsjournal.org/> , but this copy is not to be cited. >>> >>> * As mentioned by Ian Horrocks, "Articles on >>> the preprint server are >>> post-review, and differ from the published >>> version only w.r.t. >>> formatting." [Pending proof] >>> >>> * Authors can choose to give Elsevier >>> exclusive rights to publish and >>> sell their work (to libraries, individuals..), or >>> authors can pay the >>> article processing charge (APC) to make the >>> works accessible for free >>> from Elsevier's access point. In the case of >>> APC, there is no constraint >>> for Elsevier to omit existing charges to >>> libraries for those works. >>> >>> * www.websemanticsjournal.org >>> <http://www.websemanticsjournal.org/> and its archive (ie. the preprint >>> server) >>> is not funded by Elsevier, but instead funded >>> by public funds. >>> >>> * Research objects (eg underlying data, tools) >>> are not accessible, ie. >>> not hosted by www.websemanticsjournal.org >>> <http://www.websemanticsjournal.org/> or Elsevier. >>> >>> * Article contributions to JWS may only use >>> non native Web technologies >>> eg LaTeX/Word.. >>> >>> * No JWS Editor to date received payment >>> from Elsevier for their role. >>> >>> * Peer-reviews are carried out by the >>> community as opposed to Elsevier. >>> Reviewers are not paid by Elsevier (or other >>> for-profit). Reviewers may >>> be publicly funded through their academic >>> institutions or labs to carry >>> out the review process. >>> >>> * Peer-reviews for accepted and rejected >>> works are not accessible by public. >>> >>> >>> Corrections and additional information is >>> most welcome. >>> >>> >>> -Sarven >>> http://csarven.ca/#i >>> >>> >>> >>> >>> >>> ---- >>> Ivan Herman, W3C >>> Publishing@W3C Technical Lead >>> Home: http://www.w3.org/People/Ivan/ >>> mobile: +31-641044153 >>> ORCID ID: http://orcid.org/0000-0003-0782-2704 >>> >>> >>> >>> >>> >>> >>> -- >>> Krzysztof Janowicz >>> >>> Geography Department, University of California, Santa Barbara >>> 4830 Ellison Hall, Santa Barbara, CA 93106-4060 >>> >>> Email: jano@geog.ucsb.edu <mailto:jano@geog.ucsb.edu> >>> Webpage: http://geog.ucsb.edu/~jano/ >>> Semantic Web Journal: http://www.semantic-web-journal.net >> >
Received on Sunday, 13 August 2017 08:46:24 UTC