W3C home > Mailing lists > Public > semantic-web@w3.org > April 2015

ANN: WebDataCommons releases 20.4 billion quads RDFa, Microdata and Microformat data originating from 2.7 million pay-level-domains

From: Robert Meusel <robert@informatik.uni-mannheim.de>
Date: Mon, 13 Apr 2015 10:30:48 +0200
Message-ID: <552B7EB8.2060003@informatik.uni-mannheim.de>
To: semantic-web@w3.org
Hi all,

we are happy to announce a new release of the WebDataCommons RDFa,
Microdata, and Microformat data sets.

The data sets have been extracted from the December 2014 version of the
Common Crawl covering 2.01 billion HTML pages which originate from 15.7
million websites (pay-level domains).

Altogether we discovered structured data within 620 million HTML pages out
of the 2.04 billion pages contained in the crawl (30%). These pages
originate from 2.7 million different pay-level-domains out of the 15.7
million pay-level domains covered by the crawl (17%).

Approximately 571 thousand of these websites use RDFa, while 819 thousand
websites use Microdata. Microformats are used by over 1 million websites 
the crawl.


More and more websites embed structured data describing for instance
products, people, organizations, places, events, reviews, and cooking
recipes into their HTML pages using markup formats such as RDFa, Microdata
and Microformats.

The WebDataCommons project extracts all Microformat, Microdata and RDFa
data from the Common Crawl web corpus, the largest and most up-to-data web
corpus that is available to the public, and
provides the extracted data for download. In addition, we publish statistics
about the adoption of the different markup formats as well as the
vocabularies that are used together with each format.

General information about the WebDataCommons project is found at


Data Set Statistics:

Basic statistics about the December 2014 RDFa, Microdata, and Microformat
data sets as well as the vocabularies that are used together with each
markup format are found at:


Comparing the statistics to the statistics about the November 2013 
release of the data sets


we see that the adoption of the Microdata markup syntax has again
increased (819 thousand websites in 2014 compared to 463 thousand in 2013,
where both crawls cover a comparable number of websites). Where the
deployment of RDFa and Microformats is more or less stable.

Looking at the adoption of different vocabularies, we see that webmasters
mostly follow the recommendation by Google, Microsoft, Yahoo!, and Yandex to
use the schema.org vocabularies as well as their predecessors in the context
of Microdata. In the context of RDFa, the most widely used vocabulary is the
Open Graph Protocol recommended by Facebook.

Topic-wise the trend, which was already identified from 2012 to 2013
continues. We see that beside of navigational, blog and CMS related
meta-information many websites markup e-commerce related data
(Products, Offers, and Reviews) as well as contact information
(LocalBusiness, Organization, PostalAddress).


The overall size of the December 2014 RDFa, Microdata, and Microformat data
sets is 20.4 billion RDF quads. For download, we split the data into 3,533
files with a total size of 357 GB.


In addition, we have created for over 50 different schema.org classes 
separate files, including all quads from pages, deploying at least once 
the specific class.


Lots of thanks to:

+ the Common Crawl project for providing their great web crawl and thus
enabling the Web Data Commons project.
+ the Any23 project for providing their great library of structured data
+ Amazon Web Services in Education Grant for supporting WebDataCommons.

Have fun with the new data set.

Robert Meusel, Anna Primpeli, and Christian Bizer

Robert Meusel
Chair of Information Systems V
Web-based Systems Group
Universit├Ąt Mannheim
B6, 26, Room C1.04
D-68159 Mannheim
Phone: +49 621 181 2648
Mail: robert@informatik.uni-mannheim.de
Web: dws.informatik.uni-mannheim.de
Received on Monday, 13 April 2015 08:31:07 UTC

This archive was generated by hypermail 2.4.0 : Tuesday, 5 July 2022 08:45:42 UTC