Re: Schema.org sameAs/Article Test on Wikipedia

Hi Aaron - thanks for the messages. Just wanted to note receipt. We'll
reply probably sometime next week, post the US Thanksgiving holiday weekend.

Thanks again!
-Adam

On Tue, Nov 20, 2018 at 1:50 PM Aaron Bradley <aaranged@gmail.com> wrote:

> And sorry for the serial thread, but a couple of follow-up questions that
> arose after I poked around a bit more:
>
>    - You say "we're A/B testing the sameAs property on Wikipedia articles
>    having a corresponding Wikidata entry".  When there is no Wikidata entry
>    are you omitting schema.org/Article markup altogether, or are you
>    adding Article markup by omitting only the sameAs declaration?  If the
>    former, this is a pretty critical difference.  That is, comparing the
>    search relevance of an article marked up with Article (e.g.
>    https://en.wikipedia.org/wiki/World_War_II) with one without Article
>    markup (e.g. https://en.wikipedia.org/wiki/Violet_Friend) is a lot
>    different than a narrow test of the use of sameAs.  If this is the case
>    (comparison of an article with schema.org/Article, including sameAs,
>    with an article without any Article markup at all) then I'm sure this would
>    generate some interesting data to look at, but I'd be at a loss to see how
>    you could assess the impact of sameAs in this case - a viable A/B here
>    would of course require compariing articles with Article including the
>    sameAs property and articles with Article by excluding the sameAs
>    property.  But maybe I've just failed to find any instances of the latter
>    (Article markup without sameAs).
>    - Where an article is encoded with Article and the sameAs property,
>    are you in all cases (as per the examples I've been able to uncover) also
>    providing this Wikidata URI as the value for the mainEntity property?
>
> Thanks again!
>
> On Tue, Nov 20, 2018 at 11:24 AM Aaron Bradley <aaranged@gmail.com> wrote:
>
>> Very interesting Adam.  Curious is you can share with us any further
>> detail regarding *specifically* what's being tested, and how results are
>> being judged.
>>
>> That is, you say:
>> "One motivation for this work is to explore the effect of the changes on
>> search relevance. More generally, we're also interested in how structured
>> markup like this on Wikimedia content projects might be beneficial to the
>> semantic web and machine readability..."
>>
>> With regard to "the effect of the changes on search relevance", do you
>> mean how the additional markup impacts the visibility of Wikipedia pages in
>> enterprise search engines such as Google, Bing and Yandex, or are there
>> other search environments implicated here?  And what is/are the measurement
>> protocol/protocols you're using to assess "relevance" (e.g. changes in
>> search engine ranking, changes in the way a Wikipedia page is presented in
>> the search results, changes in what information is about a
>> Wikipedia-referenced entity is represented in a Google Knowledge Panel or
>> Bing Snapshot)?
>>
>> And a similar question in regard to your interest in "how markup like
>> this ... might be beneficial to the semantic web and machine
>> readability...."  How are you assessing this impact (asking because, as
>> with "search relevance", any A/B test is predicated on begin able to
>> measure the performance of A compared to B, which in turn requires that a
>> measurement protocol be in place).
>>
>> Thanks in advance for any further detail  you can provide.
>>
>> Aaron Bradley
>> Knowledge Graph Strategist
>> Electronic Arts
>>
>>
>> On Tue, Nov 20, 2018 at 5:30 AM Adam Baso <abaso@wikimedia.org> wrote:
>>
>>> Hello -
>>>
>>> I'm Adam Baso, an engineering director at the Wikimedia Foundation. We
>>> wanted to let the Schema.org community know that we're A/B testing the
>>> sameAs property on Wikipedia articles having a corresponding Wikidata
>>> entry. We're running the test on most [1] Wikipedias, including English
>>> Wikipedia. You can see this in effect for the markup on the English
>>> article for War of the Polish Succession
>>> <https://search.google.com/structured-data/testing-tool#url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FWar_of_the_Polish_Succession>
>>> .
>>>
>>> We have applied the new property to 50% of Wikipedia pages with
>>> corresponding Wikidata entries by way of the Article type. Our CDN is being
>>> populated with this change so you should see more of this markup. We'll be
>>> running this A/B test for about two months and will be evaluating results
>>> along the way.
>>>
>>> One motivation for this work is to explore the effect of the changes on
>>> search relevance. More generally, we're also interested in how structured
>>> markup like this on Wikimedia content projects might be beneficial to the
>>> semantic web and machine readability, much along the lines of other
>>> Wikimedia movement initiatives such as Structured Data on Commons
>>> <https://commons.wikimedia.org/wiki/Commons:Structured_data>.
>>>
>>> Any feedback you have on our implementation in this A/B test would be
>>> most welcome. Please share your feedback with Olga Vasileva (
>>> ovasileva@wikimedia.org). Thanks!
>>>
>>> Regards,
>>>
>>> Adam Baso
>>>
>>> Engineering Director
>>>
>>> Wikimedia Foundation
>>>
>>> [1]
>>>
>>> The following wikis are excluded from A/B testing due to overlap with
>>> other A/B tests:
>>>
>>> Indonesian: idwiki <https://id.wikipedia.org/wiki/Halaman_Utama>
>>>
>>> Portuguese: ptwiki
>>> <https://pt.wikipedia.org/wiki/Wikip%C3%A9dia:P%C3%A1gina_principal>
>>>
>>> Punjabi: pawiki
>>> <https://pa.wikipedia.org/wiki/%E0%A8%AE%E0%A9%81%E0%A9%B1%E0%A8%96_%E0%A8%B8%E0%A8%AB%E0%A8%BC%E0%A8%BE>
>>>
>>> Dutch: nlwiki <https://nl.wikipedia.org/wiki/Hoofdpagina>
>>>
>>> Korean: kowiki
>>> <https://ko.wikipedia.org/wiki/%EC%9C%84%ED%82%A4%EB%B0%B1%EA%B3%BC:%EB%8C%80%EB%AC%B8>
>>>
>>> Bhojpuri: bhwiki <https://bh.wikipedia.org>
>>>
>>> Cherokee: chrwiki
>>> <https://chr.wikipedia.org/wiki/%E1%8E%A4%E1%8E%B5%E1%8E%AE%E1%8E%B5%E1%8F%8D%E1%8F%97>
>>>
>>> Kazakh: kkwiki
>>> <https://kk.wikipedia.org/wiki/%D0%91%D0%B0%D1%81%D1%82%D1%8B_%D0%B1%D0%B5%D1%82>
>>>
>>> Catalan: cawiki <https://ca.wikipedia.org/wiki/Portada>
>>>
>>> French: frwiki
>>> <https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Accueil_principal>
>>>
>>> Yoruba: yowiki
>>> <https://yo.wikipedia.org/wiki/Oj%C3%BAew%C3%A9_%C3%80k%E1%BB%8D%CC%81k%E1%BB%8D%CC%81>
>>> Kalmyk: xalwiki
>>> <https://xal.wikipedia.org/wiki/%D0%9D%D2%AF%D1%80_%D1%85%D0%B0%D0%BB%D1%85>
>>>
>>