- From: Robert Meusel <robert@informatik.uni-mannheim.de>
- Date: Mon, 13 Apr 2015 10:29:48 +0200
- To: public-vocabs@w3.org
Hi all, we are happy to announce a new release of the WebDataCommons RDFa, Microdata, and Microformat data sets. The data sets have been extracted from the December 2014 version of the Common Crawl covering 2.01 billion HTML pages which originate from 15.7 million websites (pay-level domains). Altogether we discovered structured data within 620 million HTML pages out of the 2.04 billion pages contained in the crawl (30%). These pages originate from 2.7 million different pay-level-domains out of the 15.7 million pay-level domains covered by the crawl (17%). Approximately 571 thousand of these websites use RDFa, while 819 thousand websites use Microdata. Microformats are used by over 1 million websites within the crawl. Background: More and more websites embed structured data describing for instance products, people, organizations, places, events, reviews, and cooking recipes into their HTML pages using markup formats such as RDFa, Microdata and Microformats. The WebDataCommons project extracts all Microformat, Microdata and RDFa data from the Common Crawl web corpus, the largest and most up-to-data web corpus that is available to the public, and provides the extracted data for download. In addition, we publish statistics about the adoption of the different markup formats as well as the vocabularies that are used together with each format. General information about the WebDataCommons project is found at http://webdatacommons.org/ Data Set Statistics: Basic statistics about the December 2014 RDFa, Microdata, and Microformat data sets as well as the vocabularies that are used together with each markup format are found at: http://webdatacommons.org/structureddata/2014-12/stats/stats.html Comparing the statistics to the statistics about the November 2013 release of the data sets http://webdatacommons.org/structureddata/2013-11/stats/stats.html we see that the adoption of the Microdata markup syntax has again increased (819 thousand websites in 2014 compared to 463 thousand in 2013, where both crawls cover a comparable number of websites). Where the deployment of RDFa and Microformats is more or less stable. Looking at the adoption of different vocabularies, we see that webmasters mostly follow the recommendation by Google, Microsoft, Yahoo!, and Yandex to use the schema.org vocabularies as well as their predecessors in the context of Microdata. In the context of RDFa, the most widely used vocabulary is the Open Graph Protocol recommended by Facebook. Topic-wise the trend, which was already identified from 2012 to 2013 continues. We see that beside of navigational, blog and CMS related meta-information many websites markup e-commerce related data (Products, Offers, and Reviews) as well as contact information (LocalBusiness, Organization, PostalAddress). Download: The overall size of the December 2014 RDFa, Microdata, and Microformat data sets is 20.4 billion RDF quads. For download, we split the data into 3,533 files with a total size of 357 GB. http://webdatacommons.org/structureddata/2014-12/stats/how_to_get_the_data.html In addition, we have created for over 50 different schema.org classes separate files, including all quads from pages, deploying at least once the specific class. http://webdatacommons.org/structureddata/2014-12/stats/schema_org_subsets.html Lots of thanks to: + the Common Crawl project for providing their great web crawl and thus enabling the Web Data Commons project. + the Any23 project for providing their great library of structured data parsers. + Amazon Web Services in Education Grant for supporting WebDataCommons. Have fun with the new data set. Cheers, Robert Meusel, Anna Primpeli, and Christian Bizer -- Robert Meusel Chair of Information Systems V Web-based Systems Group Universität Mannheim B6, 26, Room C1.04 D-68159 Mannheim Phone: +49 621 181 2648 Mail: robert@informatik.uni-mannheim.de Web: dws.informatik.uni-mannheim.de
Received on Monday, 13 April 2015 08:30:08 UTC