This document provides best practices related to the
publication and usage of data on the Web designed to help support a
self-sustaining ecosystem. Data should be discoverable and
understandable by humans and machines. Where data is used in some way,
whether by the originator of the data or by an external party, such
usage should also be discoverable and the efforts of the data publisher
recognized. In short, following these best practices will facilitate
interaction between publishers and consumers.
This version of the document shows its expected scope and future
direction. A template is used to show the "what", "why" and "how" of
each best practice. Comments are sought on the usefulness of this
approach and the expected scope of the final document.
Introduction
The best practices described below have been developed to encourage and
enable the continued expansion of the Web as a medium for the exchange
of data. The growth of open data by governments across the world
[[OKFN-INDEX]], the increasing publication of research data encouraged
by organizations like the Research Data Alliance [[RDA]], the harvesting
and analysis of social media, crowd-sourcing of information, the
provision of important cultural heritage collections such as at the
Bibliothèque nationale de France [[BNF]] and the sustained growth in the
Linked Open Data Cloud [[LODC]], provide some examples of this
phenomenon.
In broad terms, data publishers aim to share data either openly or with
controlled access. Data consumers (who may also be producers themselves)
want to be able to find and use data, especially if it is accurate,
regularly updated and guaranteed to be available at all times. This
creates a fundamental need for a common understanding between data
publishers and data consumers. Without this agreement, data publishers'
efforts may be incompatible with data consumers' desires.
The openness and flexibility of the Web creates new challenges for data
publishers and data consumers, such as how to
represent, describe and make data available in a way that it will be
easy to find and to understand. In contrast to conventional databases,
for example, where there is a single data model to represent the data
and a database management system (DBMS) to control data access, data on
the Web allows for the existence of multiple ways to represent and to
access data. For more details about the challenges see the section Data on the Web Challenges.
In this context, it becomes crucial to
provide guidance to publishers that will improve consistency in the way
data is managed, thus promoting the reuse of data and also to foster
trust in the data among developers, whatever technology they choose to
use, increasing the potential for genuine innovation.
A general best practice to publish Data on the Web is to use standards. Different types of organizations specify standards that are specific to the publishing of datasets related to particular domains/applications, involving communities of users interested in that data. These standards define a common way of communicating information among the users of these communities. For example, publishing of timetables have two standards, the General Transit Feed Specification [[GTFS]] and the Service Interface for Real Time Information [[SIRI]], that specify, in a mixed way, standardize terms, standardized data formats and standardized data access. As was said before, the Best Practices proposed in this document serve a general purpose of publishing and using Data on the Web. These Best Practices are domain/application independent. They could be extended or complemented by other best practices documents or standards that cover the publishing of datasets in a more restricted universe.
Taking that into account, this document sets out a series of best practices that will help publishers and consumers face the new challenges and opportunities posed by data on the Web. They intend to serve a general purpose of publishing and using Data on the Web, but they may be specialized according to specific domains, such as Spatial Data on the Web Best Practices [[SDW]].
Best practices cover different aspects related to data publishing and
consumption, like data formats, data access, data identifiers and
metadata. In order to delimit the scope and elicit the required features
for Data on the Web Best Practices, the DWBP
working group compiled a set of use cases [[UCR]] that represent
scenarios of how data is commonly published on the Web and how it is
used. The set of requirements derived from these use cases were used to
guide the development of the best practices.
The Best Practices proposed in this document are intended to serve a
more general purpose than the practices suggested in Best Practices for
Publishing Linked Data [[LD-BP]] since it is domain-independent and
whilst it recommends the use of Linked Data, it also promotes best
practices for data on the Web in formats such as CSV [[RFC4180]] and
JSON [[RFC4627]]. The Best Practices related to the use of vocabularies
incorporate practices that stem from Best Practices for Publishing
Linked Data where appropriate.
In order to encourage data publishers to adopt the DWBP, benefits were set: comprehension; processability; discoverability; reuse; trust; linkability; access; and interoperability. They are described and related to the Best Practices in the section Best Practices Benefits.
Audience
This document provides best practices to those who publish data on the
Web. The best practices are designed to meet the needs of information
management staff, developers, and wider groups such as scientists
interested in sharing and reusing research data on the Web. While data
publishers are our primary audience, we encourage all those engaged in
related activities to become familiar with it. Every attempt has been
made to make the document as readable and usable as possible while still
retaining the accuracy and clarity needed in a technical specification.
Readers of this document are expected to be familiar with some
fundamental concepts of the architecture of the Web [[WEBARCH]], such as
resources and URIs, as well as a number of data formats. The normative
element of each best practice is the intended outcome.
Possible implementations are suggested and, where appropriate, these
recommend the use of a particular technology such as CSV, JSON and RDF.
A basic knowledge of vocabularies and data models would be helpful to
better understand some aspects of this document.
Scope
This document is concerned solely with best practices that:
are specifically relevant to data published on the Web;
encourage publication or reuse of data on the Web;
can be tested by machines, humans or a combination of the two.
As noted above, whether a best practice has or has not been followed
should be judged against the intended outcome, not the possible
approach to implementation which is offered as guidance. A best
practice is always subject to improvement as we learn and evolve the Web
together.
Context
In general, the Best Practices proposed for publication and usage of
Data on the Web refer to datasets and
distributions. Data is published in different distributions, which
is a specific physical form of a dataset. By data, "we mean known facts
that can be recorded and that have implicit meaning" [[Navathe]]. These
distributions facilitate the sharing of data on a large scale, which
allows datasets to be used for several groups of
data consumers , without regard to purpose, audience, interest,
or license. Given this heterogeneity and the fact that data publishers
and data consumers may be unknown to each other, it is necessary to
provide some information about the datasets which may also contribute to
trustworthiness and reuse, such as: structural metadata, descriptive
metadata, access information, data quality information, provenance
information, license information and usage information.
Other important aspect of publishing and sharing data on the Web
concerns the architectural bases of the Web as discussed in [[WEBARCH]].
The DWBP document
is mainly interested on the Identification principle that says that URIs
should be used to identify resources. In our context, a resource may be
a whole dataset or a specific item of given dataset. All resources
should be published with stable URIs, so that they can be referenced and
make links, via URI, between two or more resources.
The following diagram illustrates the dataset composition (data values
and metadata) together with other components related to the dataset
publication and usage. Data values correspond to the data itself and may
be available in one or more distributions, which should be defined by
the publisher considering data consumer's expectations. The Metadata
component corresponds to the additional information that describes the
dataset and dataset distributions, helping the manipulation and the
reuse of the data. In order to allow an easy access to the dataset and
its corresponding distributions, multiple Dataset Access mechanisms
should be available. Finally, to promote the interoperability among
datasets it is important to adopt Data Vocabularies and Standards.
Namespaces
The following namespace prefixes are used throughout this document.
Namespaces used in the document
Prefix
Namespace IRI
cnt
http://www.w3.org/2011/content
dcat
http://www.w3.org/ns/dcat#
dct
http://purl.org/dc/terms/
dqv
http://www.w3.org/ns/dqv#
duv
http://www.w3.org/ns/duv#
foaf
http://xmlns.com/foaf/0.1/
oa
http://www.w3.org/ns/oa#
owl
http://www.w3.org/2002/07/owl#
pav
http://pav-ontology.github.io/pav/
prov
http://www.w3.org/ns/prov#
rdf
http://www.w3.org/1999/02/22-rdf-syntax-ns#
rdfs
http://www.w3.org/2000/01/rdf-schema#
skos
http://www.w3.org/2004/02/skos/core#
Best Practices Template
This section presents the template used to describe Data on the Web
Best Practices.
Best Practice Template
Short description of the BP
Why
This section answers two crucial questions:
Why this is unique to publishing or reusing data on the Web?
How does this encourages publication or reuse of data on the
Web?
A full text description of the problem
addressed by the best practice may also be provided. It can be any
length but is likely to be no more than a few sentences.
Intended Outcome
What it should be possible to do when a data publisher follows the
best practice.
Possible Approach to Implementation
A description of a possible implementation strategy is provided.
This represents the best advice available at the time of writing but
specific circumstances and future developments may mean that
alternative implementation methods are more appropriate to achieve
the intended outcome.
How to Test
Information on how to test the BP has been met. This might or might
not be machine testable.
A benefit represents an improvement in the way how datasets are available on the Web. A Best Practice can have one or more benefits.
Reuse
Comprehension
Linkability
Discoverability
Trust
Access
Interoperability
Processability
The Best Practices
This section contains the best practices to be used by data publishers in order to help them and data consumers to overcome the different challenges faced when publishing and consuming data on the Web. One or more best practices were proposed for each one of the challenges, which are described in the section Data on the Web Challenges.
Each BP is related to one or more requirements from the Data on the Web Best Practices Use Cases & Requirements document document and the development of Data on the Web Best Practices was guided by these requirements, in such a way that each best practice should have at least one of these requirements as an evidence of its relevance.
Basic Example
John works for the Transport Agency of MyCity and he is in charge
of the publication of data on the Web about bus stops as well
as real time data about the traffic of the city. John decides to
create two datasets: one for the bus stops and other one to offer
real time traffic data.
When necessary RDF examples in Turtle syntax will be used to show the result of the
application of some best practices.
Metadata
The Web is an open information space, where the absence of a specific
context, such a company's internal information system, means that the
provision of metadata is a fundamental requirement. Data will not be
discoverable or reusable by anyone other than the publisher if
insufficient metadata is provided. Metadata provides additional
information that helps data consumers better understand the meaning of
data, its structure, and to clarify other issues, such as rights and
license terms, the organization that generated the data, data quality,
data access methods and the update schedule of datasets.
Metadata can be used to help tasks such as dataset discovery and reuse,
and can be assigned considering different levels of granularity from a
single property of a resource to a whole dataset, or all datasets from a
specific organization.
Metadata can be of different types. These types can be classified in
different taxonomies, with different grouping criteria. For example, a
specific taxonomy could define three metadata types according to
descriptive, structural and administrative features. Descriptive
metadata serves to identify a dataset, structural metadata serves to
understand the structure in which the dataset is distributed and
administrative metadata serves to provide information about the
version, update schedule etc. A different taxonomy could define
metadata types with a scheme according to tasks where metadata are
used, for example, discovery and reuse.
Provide metadata
Metadata must be provided for both human users
and computer applications.
Why
Providing metadata is a fundamental requirement when publishing
data on the Web because data publishers and data consumers may be
unknown to each other. Then, it is essential to provide information
that helps human users and computer applications to understand the
data as well as other important aspects that describes a dataset
or a distribution.
Intended Outcome
The metadata should be coherent with the described resource (i.e. dataset, distribution or the data itself) and it must be possible for humans to interpret the metadata, which makes it human-readable metadata. It also should be possible for computer applications, notably user agents, to process the metadata, which makes it machine-readable metadata.
Possible Approach to Implementation
Possible approaches to provide human readable metadata:
to provide metadata as part of an HTML Web page
to provide metadata as a separate text file
Possible approaches to provide machine readable metadata:
machine readable metadata may be provided in a serialization
format such as Turtle and JSON, or it can be embedded in the
HTML page using [[HTML-RDFA]] or [[JSON-LD]]. If multiple
formats are published separately, they should be served from the
same URL using content negotiation. Maintenance of multiple
formats is best achieved by generating each available format on
the fly based on a single source of the metadata.
when defining machine readable metadata, reusing existing
standard terms and popular vocabularies are strongly
recommended. For example, Dublin Core Metadata (DCMI) terms
[[DC-TERMS]] and Data Catalog Vocabulary [[VOCAB-DCAT]] should
be used to provide descriptive metadata.
How to Test
Check if there are coherent metadata available, both human and machine readable, for the resource.
Check if the metadata is available in a valid machine readable format, without syntax error and processable by machine.
The overall features of datasets and
distributions must be described by metadata.
Why
Explicitly providing dataset descriptive information allows user
agents to automatically discover datasets available on the Web and
it allows humans to understand the nature of the dataset and its
distributions.
Intended Outcome
Humans should be able to interpret the nature of the dataset and its distributions and user agents should be able to automatically discover datasets and distributions.
Possible Approach to Implementation
Descriptive metadata can include the following overall features
of a dataset:
The title and a description
of the dataset.
The keywords describing the dataset.
The date of publication of the dataset.
The entity responsible (publisher) for
making the dataset available.
The contact point of the dataset.
The spatial coverage of the dataset.
The temporal period that the dataset
covers.
The themes/categories covered by a
dataset.
Descriptive metadata can include the following overall features
of a distribution:
The title and a description
of the distribution.
The date of publication of the distribution.
The media type of the distribution.
The machine readable version of the descriptive metadata can be
provided using the vocabulary recommended by W3C to
describe datasets, i.e. the Data Catalog Vocabulary
[[VOCAB-DCAT]]. This provides a framework in which datasets can be
described as abstract entities.
How to Test
Check that the metadata for the dataset itself includes the
overall features of the dataset.
Check if a user agent can automatically discover the dataset.
Information about locale parameters (date,
time, and number formats, language) should be described by metadata.
Why
Providing locale parameters metadata
helps human users and computer applications to understand and to
manipulate the data, improving the reuse of the data. Providing
information about the locality for which the data is currently
published aids data users in interpreting its meaning. Date, time,
and number formats can have very different meanings, despite
similar appearances. Making the language explicit allows users to
determine how readily they can work with the data and may enable
automated translation services.
Intended Outcome
It should be possible for human users and computer applications
to interpret the meaning of dates, times and numbers accurately by
referring to locale information.
Possible Approach to Implementation
Locale parameters metadata should include the following
information:
The language(s) of the dataset.
The formats used for numeric values, dates and time.
The machine readable version of the discovery metadata may be
provided according to the vocabulary recommended by W3C to
describe datasets, i.e. the Data Catalog Vocabulary
[[VOCAB-DCAT]].
How to Test
Check that the metadata for the dataset itself includes the
language in which it is published and that all numeric, date, and
time fields have locale metadata provided either with each field
or as a general rule.
Information about the schema and internal
structure of a distribution must be described by metadata.
Why
Providing information about the internal structure of a
distribution can be helpful when exploring or querying the
dataset. Besides, structural metadata provides information that
helps to understand the meaning of the data.
Intended Outcome
Humans should be able to interpret the schema of a dataset as well as software agents should be able to automatically process the structural metadata of the dataset
distributions.
Possible Approach to Implementation
Human readable strucutral metadata usually provides the properties or columns of the dataset schema.
Machine readable structural metadata is available according to the format of a
specific distribution and it may be provided within separate
documents or embedded into the document. For more details see the
links below.
A license is a very useful piece of
information to be attached to data on the Web. According to the type
of license adopted by the publisher, there might be more or fewer
restrictions on sharing and reusing data. In the context of data on
the Web, the license of a dataset can be specified within the data, or
outside of it, in a separate document to which it is linked.
Provide data license
information
Data license information should be available.
Why
The presence of license information is essential for data
consumers to assess the usability of data. User agents, for
example, may use the presence/absence of license information as a
trigger for inclusion or exclusion of data presented to a
potential consumer.
Intended outcome
It should be possible for humans to understand possible
restrictions placed on the use of a distribution.
It should be possible for machines to automatically detect the
data license of a distribution.
Possible Approach to Implementation
Data license information can be provided as a link to a human-readable license or as a link/embedded machine-readable license.
The machine readable version of the data license metadata may be
provided using one of the following vocabularies that include
properties for linking to a license:
Dublin Core [[DC-TERMS]] (dct:license)
Creative Commons [[CC-VOCAB]] (cc:license)
schema.org [[SCHEMA-ORG]] (schema:license)
XHTML [[XHTML-VOCAB]] (xhtml:license)
There are also a number of machine readable rights languages,
including:
The Creative Commons Rights Expression Language [[ccREL]]
The Open Digital Rights Language [[ODRL]]
The Open Data Rights Statement Vocabulary [[ODRS]]
How to Test
Check that the metadata for the dataset itself includes the data
license information.
Check if a user agent can automatically detect the data license
of the dataset.
Data provenance becomes particularly
important when data is shared between collaborators who might not have
direct contact with one another either due to proximity or because the
published data outlives the lifespan of the data provider projects or
organizations.
The Web brings together business, engineering, and scientific
communities creating collaborative opportunities that were previously
unimaginable. The challenge in publishing data on the Web is providing
an appropriate level of detail about its origin. The
data producer may not necessarily be the data provider and so
collecting and conveying this corresponding metadata is particularly
important. Without provenance, consumers have no inherent way to trust
the integrity and credibility of the data being shared. Data
publishers in turn need to be aware of the needs of prospective
consumer communities to know how much provenance detail is
appropriate.
Provide data
provenance information
Data provenance information should be
available.
Why
Without accessible data provenance, data consumers will not know
the origin or history of the published data.
Intended Outcome
It should be possible for humans to know the origin or history of
the dataset.
It should be possible for machines to automatically process the
provenance information about the dataset.
Possible Approach to Implementation
The machine readable version of the data provenance can be
provided using an ontology recommended to describe provenance information, such as W3C's Provenance Ontology [[PROV-O]].
How to Test
Check that the metadata for the dataset itself includes the
provenance information about the dataset.
Check if a computer application can automatically process the
provenance information about the dataset.
The quality of a dataset can have a big impact on the quality of applications that use it. As a consequence, the inclusion of data quality considerations in data publishing and consumption pipelines is of primary importance. Usually, the assessment of quality involves
different kinds of quality dimensions, each representing groups of
characteristics that are relevant to publishers and consumers.
Measures and metrics are defined to assess the quality for each
dimension [[DQV]]. There are heuristics designed to fit specific
assessment situations that rely on quality indicators, namely, pieces
of data content, pieces of data meta-information, and human ratings
that give indications about the suitability of data for some intended
use.
Provide data quality
information
Data Quality information should be available.
Why
Data quality might seriously affect the suitability of data for
specific applications, including applications very different from
the purpose for which it was originally generated. Documenting
data quality significantly eases the process of datasets
selection, increasing the chances of reuse. Independently from
domain-specific peculiarities, the quality of data should be
documented and known quality issues should be explicitly stated in
metadata.
Intended Outcome
It should be possible for humans to have access to information
that describes the quality of the dataset and its distributions.
It should be possible for machines to automatically process the
quality information about the dataset and its distributions.
Possible Approach to Implementation
The machine readable version of the dataset quality metadata may
be provided according to the vocabulary that is being developed by
the DWBP
working group , i.e., the Data Quality Vocabulary [[DQV]].
How to Test
Check that the metadata for the dataset itself includes quality
information about the dataset.
Check if a computer application can automatically process the
quality information about the dataset.
Datasets published on the Web may change over time. Some datasets
are updated on a scheduled basis, and other datasets are changed as
improvements in collecting the data make updates worthwhile. In order
to deal with these changes, new versions of a dataset may be created.
Unfortunately, there is no consensus about when changes to a dataset should
cause it to be considered a different dataset altogether rather than a new version.
In the following, we present some scenarios where most publishers would agree that
the revision should be considered a new version of the existing dataset.
Scenario 1: a new bus stop is created and it should to be added to the dataset;
Scenario 2: an existing bus stop is removed and it
should be deleted from the dataset;
Scenario 3: an error was identified in one of the existing
bus stops stored in the dataset and this error must be corrected.
In general, multiple datasets that represent time series or
spatial series, e.g. the same kind of data for different regions
or for different years, are not considered multiple versions of the same
dataset. In this case, each dataset covers a different set of observations
about the world and should be treated as a new dataset. This is also the case with a dataset that
collects data about weekly weather forecasts for a given city, where
every week a new dataset is created to store data about that
specific week.
Scenarios 1 and 2 might trigger a major version, whereas Scenario 3 would likely trigger only a minor version.
But how you decide whether versions are minor or major is less important than that you avoid making
changes without incrementing the version indicator. Even for small changes, it is important to keep track of the
different dataset versions to make the dataset trustworthy. Publishers
should remember that a given dataset may be in use by one or more
data consumers, and they should take reasonable steps to inform those consumers when a new version is released.
For real-time data, an automated timestamp can serve as a version identifier.
For each dataset, the publisher should take a
consistent, informative approach to versioning, so data consumers can
understand and work with the changing data.
Provide a version
indicator
An indication of the version number or date should be
available for each dataset.
Why
Version information makes a revision of a dataset uniquely identifiable.
Uniqueness can be used by data consumers to determine whether and how data has
changed over time and to determine specifically which version of a
dataset they are working with. Good data versioning enables
consumers to understand if a newer version of a dataset is
available. Explicit versioning allows for repeatability in
research, enables comparisons, and prevents confusion. Using
unique version numbers that follow a standardized approach can
also set consumer expectations about how the versions differ.
Intended Outcome
It should be possible for data consumers to easily determine
which version of a dataset they are working with. It should be possible to determine
whether and how a given dataset differs from another of the same set.
Possible Approach to Implementation
The best method for providing versioning information
will vary according to the context; however, there are some basic
guidelines that can be followed, for example:
Include a unique version number or date as part of the metadata for
the dataset.
Use a consistent numbering scheme with a meaningful approach
to incrementing digits, such as [[SchemaVer]].
If the data is made available through an API, the URI used to
request the latest version of the data should not change as the
versions change, but it should be possible to request a specific
version through the API.
Use Memento [[RFC7089]], or components thereof, to
express temporal versioning of a dataset and to access the
version that was operational at a given datetime. The Memento
protocol aligns closely with the approach for assigning URIs to
versions that is used for W3C
specifications, described below.
The Web Ontology Language [[OWL2-QUICK-REFERENCE]] and the
Provenance, Authoring and versioning Ontology [[PAV]] provides a
number of annotation properties for version information.
How to Test
Check that a unique version number or date is provided with the
metadata describing the dataset.
A version history for the dataset should be
available.
Why
In creating applications that use data, it can be helpful to
understand the variability of that data over time. Interpreting
the data is also enhanced by an understanding of its dynamics.
Determining how the various versions of a dataset differ from each
other is typically very laborious unless a summary of the
differences is provided.
Intended Outcome
It should be possible for data consumers to understand how the
dataset typically changes from version to version and how any two
specific versions differ.
Possible Approach to Implementation
Provide a list of published versions and a description for each
version that explains how it differs from the previous version. An
API can expose a version history with a single dedicated URL that
retrieves the latest version of the complete history.
How to Test
Check that a list of published versions is available as well as a change log describing precisely how each version differs from the previous one.
Identifiers take many forms and are used extensively in every
information system. Data discovery, usage and citation on the Web
depends fundamentally on the use of HTTP (or HTTPS) URIs: globally
unique identifiers that can be looked up by dereferencing them over
the Internet [[RFC3986]]. It is perhaps worth emphasizing some key
points about URIs in the current context.
URIs are 'dumb strings', that is, they carry no semantics. Their
function is purely to identify a resource.
Although the previous point is accurate, it would be perverse for
a URI such as http://example.com/dataset.csv to return anything
other than a CSV file. Human readability is helpful.
When de-referenced (looked up), a single URI may offer the same
resource in more than one format. http://example.com/dataset may
offer the same data in, say, CSV, JSON and XML. The server returns
the most appropriate format based on
content negotiation .
One URI may redirect to another.
De-referencing a URI triggers a computer program to run on a
server so that the URI acts as a call to an API. The server may
therefore do something as simple as return a single, static file, or
it may carry out complex processing. Precisely what processing is
carried out, i.e. the software on the server, is completely
independent of the URI itself.
Use persistent
URIs as identifiers of datasets
Datasets must be identified by a persistent
URI.
Why
Adopting a common identification system enables basic data
identification and comparison processes by any stakeholder in a
reliable way. They are an essential pre-condition for proper data
management and reuse.
Developers may build URIs into their code and so it is important
that those URIs persist and that they dereference to the same
resource over time without the need for human intervention.
Intended Outcome
Datasets or information about datasets, must be discoverable and
citable through time, regardless of the status, availability or
format of the data.
Possible Approach to Implementation
To be persistent, URIs must be designed as such. This requires a
different mindset to that used when creating a Web site designed
for humans to navigate their way through. A lot has been
written on this topic, see, for example, the European Commission's
Study on Persistent URIs [[PURI]] which in turn links to many other
resources.
Where a data publisher is unable or unwilling to manage a URI
space directly for persistence, an alternative approach is to use
a redirection service such as
Permanent Identifiers for the Web
or purl.org.
These provide persistent URIs that can be redirected as required
so that the eventual location can be ephemeral. The software
behind such services is freely available so that it can be
installed and managed locally if required.
Digital Object Identifiers (DOIs)
offer a similar alternative. These identifiers are defined
independently of any Web technology but can be appended to a 'URI
stub.' DOIs are an important part of the digital infrastructure
for research data and and libraries.
How to Test
Check that each dataset in question is identified using a URI
that has been assigned under a controlled process as set out in
the previous section. Ideally, the relevant Web site includes a
description of the process and a credible pledge of persistence
should the publisher no longer be able to maintain the URI space
themselves.
Use
persistent URIs as identifiers within datasets
Datasets should use and reuse other people's
URIs as identifiers where possible.
Why
The power of the Web lies in the Network effect. The
first telephone only became useful when the second telephone meant
there was someone to call; the third telephone made both of them
more useful yet. Data becomes more valuable if it refers to other
people's data about the same thing, the same place, the same
concept, the same event, the same person, and so on. That means
using the same identifiers across datasets and making sure that
your identifiers can be referred to by other datasets. When those
identifiers are HTTP URIs, they can be looked up and more data
discovered.
These ideas are at the heart of the 5
Stars of Linked Data where one data point links to another,
and of Hypermedia
where links may be to further data or to services (or more
generally 'affordances') that act on or relate to the data in some
way. Examples include a bug reporting mechanisms, processors, a
visualization engine, a sensor, an actuator etc. In both Linked
Data and Hypermedia, the emphasis is put on the ability for
machines to traverse from one resource to another following links
that express relationships.
That's the Web of Data.
Intended Outcome
That one data item can be related to others across the Web
creating a global information space accessible to humans and
machines alike.
Possible Approach to Implementation
This is a topic in itself and a general document such as this can
only include superficial detail.
Developers know that very often the problem they're trying to
solve will have already been solved by other people. In the same
way, if you're looking for a set of identifiers for obvious things
like countries, currencies, subjects, species, proteins, cities
and regions, Nobel prize winners – someone's done it already. The
steps described for discovering
existing vocabularies [[LD-BP]] can readily be adapted.
ensure URI sets you use are published by a trusted group or
organization;
ensure URI sets have persistent URIs.
If you can't find an existing set of identifiers that meet your
needs then you'll need to create your own, following the patterns
for URI persistence so that others will add value to your data by
linking to it.
URIs can be long. In a dataset of even moderate size, storing
each URI is likely to be repetitive and obviously wasteful.
Instead, define locally unique identifiers for each element and
provide data that allows them to be converted to globally unique
URIs programmatically. The Metadata Vocabulary for Tabular Data
[[tabular-metadata]] provides mechanisms for doing this within
tabular data such as CSV files, in particular using URI
template properties such as the about
URL property.
How to Test
Check that within the dataset, references to things that don't
change or that change slowly, such as countries, regions,
organizations and people, as referred to by URIs or by short
identifiers that can be appended to a URI stub. Ideally the URIs
should resolve, however, they have value as globally scoped
variables whether they resolve or not.
URIs should be assigned to individual versions
of datasets as well as the overall series.
Why
Like documents, many datasets fall into natural series or groups.
For example:
noon temperature readings in central London 1850 to the
present day;
today's noon temperature in London;
the temperature in London at noon on 3rd June 2015.
In different circumstances, it will be appropriate to refer
separately to each of these examples (and many like them).
Intended Outcome
It should be possible to refer to a specific version of a dataset
and to concepts such as a 'dataset series' and 'the latest
version.'
Possible Approach to Implementation
The W3C provides a good example of how to do this. The
(persistent) URI for this document is
http://www.w3.org/TR/2015/WD-dwbp-20150224/. That identifier
points to an immutable snapshot of the document on the day of its
publication. The URI for the 'latest version' of this document is
http://www.w3.org/TR/dwbp/ which is an identifier for a series of
closely related documents that are subject to change over time. At
the time of publication, these two URIs both resolve to this
document. However, when the next version of this document is
published, the 'latest version' URI will be changed to point to
that.
This example needs to be rewritten.
How to Test
Check that each version of a dataset has its own URI, and that
logical groups of datasets are also identifiable.
The formats in which data is made available to consumers are a key
aspect of making that data usable. The best, most flexible access
mechanism in the world is pointless unless it serves data in formats
that enable use and reuse. Below we detail best practices in selecting
formats for your data, both at the level of files and that of
individual fields. W3C encourages use of formats that can be used by
the widest possible audience and processed most readily by computing
systems. Source formats, such as database dumps or spreadsheets, used
to generate the final published format, are out of scope. This
document is concerned with what is actually published rather than
internal systems used to generate the published data.
Use
machine-readable standardized data formats
Data must be available in a machine-readable
standardized data format that is adequate for its intended or
potential use.
Why
As data becomes more ubiquitous, and datasets become larger and
more complex, processing by computers becomes ever more crucial.
Posting data in a format that is not machine readable places
severe limitations on the continuing usefulness of the data. Data
becomes useful when it has been processed and transformed into
information.
Using non-standard data formats is costly and inefficient, and
the data may lose meaning as it is transformed. On the other hand,
standardized data formats enable interoperability as well as
future uses, such as remixing or visualization, many of which
cannot be anticipated when the data is first published. The use of
non-proprietary data formats should also be considered since it
increases the possibilities for use and reuse of data
Intended Outcome
It should be possible for machines to easily read and process
data published on the Web.
It should be possible for data consumers to use computational
tools typically available in the relevant domain to work with the
data.
It should be possible for data consumers who wants to use or
reuse the data to do so without investment in proprietary
software.
Possible Approach to Implementation
Make data available in a machine readable standardized data
format that is easily parseable including but not limited to CSV,
XML, Turtle, NetCDF, JSON and RDF.
How to Test
Check that the data format conforms to a known machine-readable
data format specification.
Data should be available in multiple data
formats.
Why
Providing data in more than one format reduces costs incurred in
data transformation. It also minimizes the possibility of
introducing errors in the process of transformation. If many users
need to transform the data into a specific data format, publishing
the data in that format from the beginning saves time and money
and prevents errors many times over. Lastly it increases the
number of tools and applications that can process the data.
Intended Outcome
It should be possible for data consumers to work with the data
without transforming it.
Possible Approach to Implementation
Consider the data formats most likely to be needed by intended
users, and consider alternatives that are likely to be useful in
the future. Data publishers must balance the effort required to
make the data available in many formats, but providing at least
one alternative will greatly increase the usability of the data.
How to Test
Check that the complete dataset is available in more than one
data format.
Data is often represented in a structured and controlled way, making
reference to a range of vocabularies, for example, by defining types
of nodes and links in a data graph or types of values for columns in a
table, such as the subject of a book, or a relationship “knows”
between two persons. Additionally, the values used may come from a
limited set of pre-existing values or resources: for example object
types, roles of a person, countries in a geographic area, or possible
subjects for books. Such vocabularies ensure a level of control,
standardization and interoperability in the data. They can also serve
to improve the usability of datasets. Say, a dataset contains a
reference to a concept described in several languages. Such reference
allows applications to localize their display of their search
depending on the language of the user.
According to W3C, vocabularies
define the concepts and relationships (also referred to as “terms” or
“attributes”) used to describe and represent an area of concern.
Vocabularies are used to classify the terms that can be used in a
particular application, characterize possible relationships, and
define possible constraints on using those terms. Several categories
of vocabularies have been coined, for example, ontology, controlled
vocabulary, thesaurus, taxonomy, code list, semantic network.
There is no strict division between the artifacts referred to by
these names. “Ontology” tends however to denote the vocabularies of
classes and properties that structure the descriptions of resources in
(linked) datasets. In relational databases, these correspond to the
names of tables and columns; in XML, they correspond to the elements
defined by an XML Schema. Ontologies are the key building blocks for
inference techniques on the Semantic Web. The first means offered by
W3C for creating ontologies is the RDF Schema [[RDF-SCHEMA]] language.
It is possible to define more expressive ontologies with additional
axioms using languages such as those in The Web Ontology Language
[[OWL2-OVERVIEW]].
On the other hand, “controlled vocabularies”, “concept schemes”,
“knowledge organization systems” enumerate and define resources that
can be employed in the descriptions made with the former kind of
vocabulary. A concept from a thesaurus, say, “architecture”, will for
example be used in the subject field for a book description (where
“subject” has been defined in an ontology for books). For defining the
terms in these vocabularies, complex formalisms are most often not
needed. Simpler models have thus been proposed to represent and
exchange them, such as the ISO 25964 data model [[ISO-25964]] or W3C's
Simple Knowledge Organization System [[SKOS-PRIMER]].
Use
standardized terms
Standardized terms should be used to provide
data and metadata.
Why
Using standardized lists of codes other commonly used terms for data
and metadata values as much as possible helps avoiding
ambiguity and clashes between these values.
Intended Outcome
The benefit of using standardized code lists and other commonly
used terms is to enhance interoperability and consensus among
data publishers and consumers.
Possible Approach to Implementation
Values in datasets should refer as much as possible to
standardization efforts or organizations, which defines terms or codes as a
clear reference.
Organizations like the Open Geospatial Consortium (OGC), ISO, W3C, libraries and research data services, etc., provide
list of codes, terminologies or even Linked Data vocabularies that can be used for this.
A key point is to make sure the dataset or its documentation provides enough (human- and machine-readable) context
for the values, so that data consumers can retrieve and exploit the standardized meaning
of these values. In the context of the Web, using unambiguous, Web-based identifiers for standardized values
is an efficient way to do this.
How to Test
Check that the terms or codes to be used are defined in a
standard organization/working group of body such as IETF, OGC,
W3C, etc.
Shared vocabularies should be used to encode
data and metadata.
Why
Use of shared vocabularies captures and facilitates consensus in communities.
Reusing existing vocabularies to encode datasets and metadata increases interoperability and reduces
redundancies, encouraging reuse of these data. In particular, the
use of shared vocabularies for metadata (especially structural, provenance, quality
and versioning metadata) helps the automatic processing of both data and metadata.
Intended Outcome
Datasets and metadata sets are easier to be compared by humans or machines when
they use the same vocabulary to describe metadata.
When two datasets or metadata sets use the same vocabulary, (automatic) processing tools
designed for one can be more easily applied to the other. This greatly facilitates re-use of datasets.
Possible Approach to Implementation
The Vocabularies section of the W3C Best Practices for
Publishing Linked Data [LD-BP] provides guidance on the
discovery, evaluation and selection of existing vocabularies.
How to Test
Using vocabulary repositories like the Linked Open Vocabularies repository
or lists or services mentioned in technology-specific best practices
such as the Best Practices for Publishing Linked Data [LD-BP],
or the Core Initial Context for RDFa and JSON-LD,
check that classes, properties, terms, elements or attributes used to represent a dataset do not replicate those
defined by vocabularies used for other datasets.
When reusing a vocabulary, a data publisher
should opt for a level of formal semantics that fit data and
applications.
Why
Formal semantics help to establish precise specifications
that support establishing the intended meaning of the vocabulary and
the performance of complex tasks such as reasoning. On the other
hand, complex vocabularies require more effort to produce and
understand, which could hamper their reuse, as well as the
comparison and linking of datasets exploiting them. Highly
formalized data is also harder to exploit by inference engines: for
example, using an OWL class in a position where a SKOS concept is
enough, or using OWL classes with complex OWL axioms raises the
formal complexity of the data according to the OWL Profiles
[[OWL2-PROFILES]]. Data producers should therefore seek to identify
the right level of formalization for particular domains, audiences
and tasks, and maybe offer different formalization levels when one
size does not fit all.
Intended Outcome
The data supports all application cases but should not be more
complex to produce and reuse than necessary;
Possible Approach to Implementation
Identify the "role" played by the vocabulary for the datasets, say,
providing classes and properties used to type resources and provide
the predicates for RDF statements, or elements in an XML Schema, as
opposed to providing simple concepts or codes that are used for
representing attributes of the resources described in a dataset.
When simpler data models are enough to convey the necessary semantics,
represent vocabularies using them.
Even when a language with rich formal semantics like OWL is used to
express a vocabulary, it is preferable that this vocabulary has a minimal
ontological commitment, i.e. by featuring only the formal axioms
that enable inferences and validation checks
that have been explicitly identified as relevant for the domain or application at hand.
The more axioms are used to specify a vocabulary, the narrower its usage is;
unnecessary axioms unnecessarily constrain the reuse of a vocabulary
across applications.
How to Test
For formal knowledge representation languages, applying an
inference engine on top of the data that uses a given vocabulary
does not produce too many statements that are unnecessary for target
applications.
To support best practices for publishing sensitive data, data
publishers should identify all sensitive data, assess the exposure risk,
determine the intended usage, data user audience and any related usage
policies, obtain appropriate approval, and determine the appropriate
security measures needed to taken to protect the data, which should also
account for secure authentication and use of HTTPS.
Data publishers should preserve the privacy of individuals where the
release of personal information would endanger safety (unintended
accidents) or security (deliberate attack). Privacy information might
include: full name, home address, mail address, national identification
number, IP address (in some cases), vehicle registration plate number,
driver's license number, face, fingerprints, or handwriting, credit card
numbers, digital identity, date of birth, birthplace, genetic
information, telephone number, login name, screen name, nickname, health
records etc.
At times, because of sharing policies sensitive data may not be
available in part or in its entirety. Data unavailability represents
gaps that may affect the overall analysis of datasets. To account for
unavailable data, data publishers should publish information about
unavoidable data gaps.
Provide
data unavailability reference
References to data that is not open, or
available under different restrictions to the origin of the reference,
should provide explanation about how the referred data can be accessed
and who can access it.
Why
Publishing online documentation about unavailable data due to
sensitivity issues provides a means for publishers to explicitly
identify knowledge gaps. This provides a contextual explanation for
consumer communities thus encouraging use of the data that is
available.
Intended Outcome
Publishers should provide information about data that is referred
to from the current dataset but that is unavailable or only
available under different conditions.
Possible Approach to Implementation
Depending on the machine/human context there are a variety of ways to indicate data unavailability. Data publishers may publish an HTML document that gives a
human-readable explanation for data unavailability. From a machine application interface perspective, appropriate HTTP status codes with customized human readable messages can be used. Examples of status codes include: 404 (file not found), 410 (permanently removed), 503 (service *providing data* unavailable).
How to Test
If the dataset includes references to other data that is
unavailable, check whether an explanation is available in the
metadata and/or description of it.
Providing easy access to data on the Web enables both humans and
machines to take advantage of the benefits of sharing data using the Web
infrastructure. By default, the Web offers access using Hypertext
Transfer Protocol (HTTP) methods. This provides access to data at an
atomic transaction level. However, when data is distributed across
multiple files or requires more sophisticated retrieval methods
different approaches can be adopted to enable data access, including
bulk download and APIs.
One approach is packaging data in bulk using non-proprietary file
formats (for example tar files). Using this approach, bulk data is
generally pre-processed server side where multiple files or directory
trees of files are provided as one downloadable file. When bulk data is
being retrieved from non-file system solutions, depending on the data
user communities, the data publisher can offer APIs to support a series
of retrieval operations representing a single transaction.
For data that is streaming to the Web in “real time” or “near real
time”, data publishers should publish data or use APIs to enable
immediate access to data, allowing access to critical time sensitive
data, such as emergency information, weather forecasting data, or
published system metrics. In general, APIs should be available to allow
third parties to automatically search and retrieve data published on the
Web.
On a further note, it can be observed that data on the Web is
essentially about the description of entities identified by a unique,
Web-based, identifier (an URI). Once the data is dumped and sent to an
institute specialised in digital preservation the link with the Web is
broken (dereferencing) but the role of the URI as a unique identifier
still remains. In order to increase the usability of preserved dataset
dumps it is relevant to maintain a list of these identifiers.
Provide bulk download
Data should be available for bulk download.
Why
When Web data is distributed across many URIs but might logically
be organized as one container, accessing the data in bulk can be useful.
Bulk access provides a consistent means to handle the data as one
container. Individually accessing data over many retrievals can be cumbersome
and, if used to reassemble the complete dataset, can lead to inconsistent approaches to handling the data.
Intended Outcome
It should be possible to download data on the Web in bulk. Data
publishers should provide a way, through either a single-file download or
a single API call, for consumers to access all the data. Large file transfers (which would require more time than a typical user would consider reasonable) should be enabled by dedicated file-transfer protocols. The bulk download should include the metadata describing the dataset. Discovery metadata [[VOCAB-DCAT]] should also be available outside the bulk downlaod.
Possible Approach to Implementation
Depending on the nature of the data and consumer needs, possible
approaches could include the following:
Preprocessing a copy of the data into a compressed archive format
and making the data accessible from one URI. This is
particularly useful for handling data that changes infrequently.
Hosting an API that includes the ability to
retrieve a bulk download in addition to dynamic queries. This
approach is useful for capturing a complete snapshot of dynamic data.
For very large datasets, bulk file transfers can be enabled via means other than http, such as bbcp or GridFTP.
How to Test
Humans can retrieve complete copies of preprocessed bulk data through
existing tools such as a browser via a single request.
If your dataset is large, enable users and applications to readily work with useful subsets of your data.
Why
Large datasets can be difficult to move from place to place. It can also be inconvenient for users to store or parse a large dataset. Users should not have to download a complete dataset if they only need a subset of it. Moreover, Web applications that tap into large datasets will perform better if their developers can take advantage of “lazy loading”, working with smaller pieces of a whole and pulling in new pieces only as needed. The ability to work with subsets of the data also enables offline processing to work more efficiently. Real-time applications benefit in particular, as they can update more quickly.
Intended Outcome
Both human users and applications should be able to access subsets of a dataset, rather than the entire thing, as needed. Subsetting approaches should aim for a high ratio of needed data to unneeded data for the largest number of users. Static datasets that users in the domain would consider to be large should be downloadable in smaller pieces. APIs should make slices or filtered subsets of the data available, the granularity depending on the needs of the domain and the demands of performance in a Web application.
Possible Approaches to Implementation
Consider the expected use cases for your dataset and determine what types of subsets are likely to be most useful. An API is usually the most flexible approach to serving subets of data, as it allows customization of what data is transferred, making the available subsets much more likely to provide the needed data--and little unneeded data--for any given situation. The granularity should be suitable for Web application access speeds. (An API call that returns within one second enables an application to deliver interactivity that feels natural. Data that takes more than ten seconds to deliver will likely cause users to suspect failure.)
Another way to subset a dataset is to simply split it into smaller units and make those units individually available for download or viewing.
It can also be helpful to mark up a dataset so that individual sections through the data (or even smaller pieces, if expected use cases warrant it) can be processed separately. One way to do that is by indicating “slices” with the RDF Data Cube Vocabulary.
How to Test
Check that the full content of the dataset can be recovered by retrieval of multiple subsets.
Use content negotiation for serving data available in multiple formats
It is recommended to use content negotiation for
serving data available in multiple formats.
Why
It is possible to have data being served in a HTML page mixed with
human-readable and machine-readable data. RDFa could be used to mix
HTML content with semantic data.
But, in some cases this page is subject of scraping by some
applications in order to get data available. When structured data is
mixed with HTML, but it is possible to have a different
representation with the same structured data, written in Turtle or
JSON-LD, it is recommended to serve this page using Content
Negotiation.
A dataset can also be served in different representations, and can be retrieved by using an API or by direct access to the resource URI. In those cases, HTTP Content Negotiation technique can be used.
Intended Outcome
It should be possible to serve the same resource with different
representations.
Possible Approach to Implementation
A possible approach to implementation is to configure the Web
server to deal with content negotiation of the requested resource.
The specific format of the resource's representation can be accessed
by the URI or by the Content-type of the HTTP Request.
How to test
Check the available representations of the resource and try to get them specifying the accepted content on the HTTP Request header.
Evidence
Relevant requirements:
Benefits
Reuse
Access
Provide real-time
access
When data is produced in real-time, it should be
available on the Web in real-time.
Why
The presence of real-time data on the Web enables access to
critical time sensitive data, and encourages the development of
real-time Web applications. Real-time access is dependent on
real-time data producers making their data readily available to the
data publisher. The necessity of providing real-time access for a
given application will need to be evaluated on a case by case basis
considering refresh rates, latency introduced by data post
processing steps, infrastructure availability, and the data needed
by consumers. In addition to making data accessible, data publishers
may provide additional information describing data gaps, data errors
and anomalies, and publication delays.
Intended Outcome
Data should be available at real time or near real time, where
real-time means a range from milliseconds to a few seconds after the
data creation, and near real time is a predetermined delay for
expected data delivery.
Possible Approach to Implementation
Real-time data accessibility may be achieved through two means:
Push - as data is produced the producers communicates data to
the data publisher either by disseminating data to the publisher
or making storage available accessible to the data producer.
On-Demand (Pull) - available real-time data is made available
upon request. In this case, data publishers will provide an API to
facilitate these read-only requests.
In addition to data access, to ensure credibility providing access to
error conditions, anomalies, and instrument "house keeping" data
enhance real-time applications ability to interpret and convey
real-time data quality to consumers.
How to Test
To adequately test real time data access, data will need to be tracked
from the time it is initially collected to the time it is published
and accessed. [[PROV-O]] can be used to describe these activities.
Caution should be used when analyzing real-time access for systems
that consist of multiple computer systems. For example, tests that
rely on wall clock time stamps may reflect inconsistencies between the
individual computer systems as opposed to data publication time
latency.
Data must be available in an up-to-date manner
and the update frequency made explicit.
Why
Data on the Web availability should closely coincide with data
provided at creation time, collection time, or after it has been
processed or changed. Carefully synchronizing data publication to
the update frequency encourages data consumer confidence and reuse.
Intended Outcome
When new data is provided or data is updated, it must be published
to coincide with the data changes.
Possible Approach to Implementation
Implement an API to enable data access. When data is provided by
bulk access, new files with new data should be provided as soon as
additional data is created or updated. Or, use technologies that are intended to expose data on the Web using interlinked resources, like Activity Streams or Atom.
If the site provides multiple data feeds providing multiple updates, the last update date time stamp should be co-located with each separate data feed. The international date format is recommended to avoid any ambiguity https://www.w3.org/International/questions/qa-date-format.
How to Test
Write test standard operating procedure for data publisher to keep
test data on Web site up to date.
Following standard operating procedure:
Write test client to access published data.
Access data and save first copy locally.
Publish an updated version of data.
Access data and save second copy locally.
Compare first copy to second copy to verify change.
In the following, we present best practices related with the creation of APIs to provide data access.
Make Data Available through an API
Offer an API to serve data if you have the resources to do so.
Why
An API offers the greatest flexibility and processability for consumers of your data. It can enable real-time data usage, filtering on request, and the ability to work with the data at an atomic level. If your dataset is large, frequently updated, or highly complex, an API is likely to be the best option for publishing your data.
Intended Outcome
Developers will have programmatic access to the data for use in their own applications. Data can be updated without requiring effort on the part of consumers.
Possible Approach to Implementation
Creating an API is a little more involved than posting data for download. It requires some understanding of how to build a Web application. One need not necessarily build from scratch, however. If you use a data management platform, such as CKAN, you may be able to enable an existing API. Many Web development frameworks include support for APIs, and there are also frameworks written specifically for building custom APIs.
How to Test
It should be possible for Web applications to obtain specific data by querying a programmatic interface. Use a test client to simulate calls and responses, making sure that the performance is acceptable.
To review the BP "Use an API" and possibly rewrite Possible Approach to Impelmentation section.
Use Web Standards as the foundation of your API
When designing APIs, it is recommended to use an architectural style that is founded on the technologies of the Web itself. Web standards such as URIs, HTTP verbs, HTTP response codes, MIME types, typed HTTP Links, and content negotiation can all help solve difficult problems and enable you to build a flexible and useful data service.
Why
APIs that are built on Web standards leverage the strengths of the Web. For example, using HTTP verbs as methods and URIs that map directly to individual resources helps to avoid tight coupling between requests and responses, making for an API that is easy to maintain and can readily be understood and used by many developers. The statelessness of the Web can be a strength in enabling quick scaling, and using hypermedia enables rich interactions with your API.
Intended Outcome
Developers who have some experience with APIs based on Web standards, such as REST, will have an initial understanding of how to use your API because it uses a standardized interface. Your API will also be easier to maintain.
Possible Approaches to Implementation
REST (REpresentational State Transfer) is an architectural style that, when used in a Web API, takes advantage of the architecture of the Web itself. A full discussion of how to build a RESTful API is beyond the scope of this document, but there are many resources and a strong community that can help in getting started. There are also many RESTful development frameworks available. If you are already using a Web development framework that supports building REST APIs, consider using that. If not, consider an API-only framework that uses REST.
Another aspect of implementation to consider is making a hypermedia API, one that responds with links rather than data alone. Links are what make the Web a web, and data APIs can be more useful and usable by including links in their responses. The links can offer additional resources, documentation, and navigation. Even for an API that does not meet all the constraints of REST, returning links in responses can make for a service that is rich and self-documenting.
How to Test
Check that the service avoids using http as a tunnel for calls to custom methods, and check that URIs do not contain method names.
Provide complete information on the Web about your API. Be sure to update documentation as you add features or make changes.
Why
Developers are the primary consumers of an API. In order to develop against it, they will need to understand how to use it. Providing comprehensive documentation in one place allows developers to code efficiently. Highlighting changes enables your users to take advantage of new features and adapt their code if needed.
Intended Outcome
The whole set of information related to the API—how to use it, notices of recent changes, contact information, and so on—should be easily browsable on the Web.
Developers should be able to obtain detailed information about each call to the API, including the parameters it takes and what it is expected to return.
The API should be self-documenting as well, so that calls return helpful information about errors and usage.
Recent changes to the API itself should be readily discoverable by users.
API users should be able to contact the maintainers with questions, suggestions, or bug reports.
Possible Approach to Implementation
A typical API reference provides a comprehensive list of the calls the API can handle, describing the purpose of each one, detailing the parameters it allows and what it returns, and giving one or more examples of its use. One nice trend in API documentation is to provide a form in which developers can enter specific calls for testing, to see what the API returns for their use case. There are now tools available for quickly creating this type of documentation, such as Swagger, io-docs, OpenApis, and others.
How to Test
Check that every call enabled by your API is described in your documentation. Make sure you provide details of what parameters are required or optional and what each call returns. The quality of documentation is also related to usage and feedback from developers. Try to get constant feedback from your users about the documentation.
Avoid changes to your API that break client code, and communicate any changes in your API to your developers when evolution happens.
Why
When developers implement a client for your API, they may rely on specific characteristics that you have built into it, such as the schema or the format of a response. Avoiding breaking changes in your API minimizes breakage to client code. Communicating changes when they do occur enables developers to take advantage of new features and, in the rare case of a breaking change, take action.
Intended Outcome
Developer code will continue to work. Developers will know of improvements you make and be able to make use of them. Breaking changes to your API will be rare, and if they occur, developers will have sufficient time and information to adapt their code. That will enable them to avoid breakage, enhancing trust.
Possible Approach to Implementation
When improving your API, focus on adding new calls or new, unrequired options rather than changing how existing calls work. Existing clients can ignore such changes and will continue functioning.
If using a fully RESTful style, you should be able to avoid changes that affect developers by keeping home resource URIs constant and changing only elements that your users do not code to directly. If you need to change your data in ways that are not compatible with the extension points that you initially designed, then a completely new design is called for, and that means changes that break client code. In that case, it’s best to implement the changes as a new REST API, with a different home resource URI.
If using an architectural style that does not allow you to make moderately significant changes without breaking client code, use versioning. Indicate the version in the response header. Version numbers should be reflected in your URIs or in request "accept" headers (using content negotiation). When versioning in URIs, include the version number as far to the left as possible. Keep the previous version available for developers whose code has not yet been adapted to the new version.
Changes to the API should be announced on your API documentation site. To notify users directly of changes, it's a good idea to create a mailing list and encourage developers to join. You can then announce changes there, and this provides a nice mechanism for feedback as well. It also allows your users to help each other.
How to Test
Release changes initially to a test version of your API before applying them to the production version. Invite developers to test their applications on the test version and provide feedback.
This section describes best practices related to data
preservation. Albeit being a closely related topic archiving
is considered out of scope for this group and therefore not covered here.
Data Preservation Best Practices are still under discussion (see ISSUE-251 ). We're keen to hear comments about the following questions:
Do we need the BP "Use a trusted serialization format for preserved data dumps" or this should be covered under the BP about using standardized formats?
Do we need the BP "Update the status of identifiers" or this should be covered under versioning or unavailability?
Do we need to rewrite the BP Assess dataset coverage to make it more clear that datasets should have minimal dependencies on external entities that may not be preserved?
Assess dataset
coverage
The coverage of a dataset should be assessed
prior to its preservation.
Why
A chunk of Web data is by definition dependent on the rest of the
global graph. This global context influences the meaning of the
description of the resources found in the dataset. Ideally, the
preservation of a particular dataset would involve preserving all
its context. That is the entire Web of Data.
At ingestion time an evaluation of the linkage of Web data dataset
dump to already preserved resources is assessed. The presence of all
the vocabularies and target resources in uses is sought in a set of
digital archives taking care of preserving Web data. Datasets for
which very few of the vocabularies used and/or resources pointed out
are already preserved somewhere should be flagged as being at risk.
Intended Outcome
It should be possible to appreciate the coverage and external
dependencies of a given dataset.
Possible Approach to Implementation
The assessment can be performed by the digital preservation
institute or the dataset depositor. It essentially consists in
checking whether all the resources used are either already preserved
somewhere or provided along with the new dataset considered for
preservation.
How to Test
Datasets making references to portions of the Web of Data which are
not preserved should receive a lower score than those using common
resources.
Use a trusted
serialisation format for preserved data dumps
Data depositors willing to send a data dump for
long term preservation must use a well established serialisation.
Why
Web data follows an abtract data model that can be expressed in
different ways (RDF/XML, JSON-LD, ...). Using a well-established
serialisation of this data increases its chances of reuse.
Institutes, such as national archives, that are engaged in digital preservation are tasked with monitoring
file formats regularly for potential risk of obsolescence. Datasets which have been acquired in some
format some years ago may have to be converted into another format
in order to still be usable with more modern software (see
[[ROSENTHAL]]). This task can be made more challenging, or even
impossible, if non-standard serialisation formats are used by data
depositors.
Intended Outcome
It should be possible to read and load the dataset into a computer for manipulation
even if the original software that was used to create it is no longer available or supported.
Possible Approach to Implementation
Give preference to non-binary Web data serialisation formats that are available as open
standards. For instance those provided by the W3C [[FORMATS]].
How to Test
Check that the dataset can be read by a standard text editor. Try to dereference the HTTP URIs present in the data dump using for example [[cURL]], confirming that the Content-Type header matches the format you expect to get.
Preserved resources should be linked through URIs with their
"live" counterparts.
Why
URI dereferencing is a primary interface to data on the Web.
Linking preserved datasets with the original URI informs the data
consumer (which might be a computer programme) that there are other, more recent, versions and facilitates determining the status of these resources.
During its life cycle a dataset may undergo several modifications resulting in multiple versions.
Although URIs assigned to things are not expected to change, the
description of these resource will evolve over time.
During this evolution, several snapshots could be made available for
preservation and accessed as earlier versions of the current dataset.
Intended Outcome
A link is maintained between the URI of a resource, the most
up-to-date description available for it, and preserved descriptions.
If the dataset resource does not exist any more then the description should say
so and refer to the last preserved description that was available.
Possible Approach to Implementation
There are a variety of HTTP status codes that could be put into use
to relate the URI with its preserved description. In particular,
200, 410 and 303 can be used for different scenarios:
200 => there is a new description which contains pointers to the
archived description
410 => the resource is no longer available but it has been
removed under a controlled process cf. 404 which simply states
that something is not available.
303 => the resource identified by this URI is no longer
served here but there is a preserved description at a different
location.
In addition to the status codes, HTTP Link headers can also be used
to relate resources to their preserved descriptions.
The following example needs to be updated to use the URI http://data.mycity.example.com/public-transport/road/bus/dataset/bus-stops-2015-05-05.
How to Test
Check that dereferencing the URI of a preserved dataset returns
information about its current status and availability.
Publishing data on the Web enables data sharing on a large scale,
providing data access to a wide range of audiences with different levels
of expertise. Data publishers want to ensure that the data published is
meeting the data consumer needs and user feedback is crucial. Feedback
has benefits for both data publishers and data consumers, helping data
publishers to improve the integrity of their published data, as well as
to encourage the publication of new data. Feedback allows data consumers
to have a voice describing usage experiences (e.g. applications using
data), preferences and needs. When possible, feedback should also be
publicly available for other data consumers to examine. Making feedback
publicly available allows users to become aware of other data consumers,
supports a collaborative environment, and allows user community
experiences, concerns or questions are currently being addressed.
From a user interface perspective there are different ways to gather
feedback from data consumers, including site registration, contact
forms, quality ratings selection, surveys and comment boxes for
blogging. From a machine perspective the data publisher can also record
metrics on data usage or information about specific applications
consumers are currently relying upon. Feedback such as this establishes
a line of communication channel between data publishers and data
consumers. In order to quantify and analyze usage feedback, it should be
recorded in a machine-readable format. Publicly
available feedback should be displayed in a human-readable form through
the user interface.
This section provides some BP to be followed by data publishers in
order to enable data consumers to provide feedback about the consumed
data. This feedback can be for humans or machines.
Gather feedback from
data consumers
Data publishers should provide a means for
consumers to offer feedback.
Why
Providing feedback contributes to improving the quality of
published data, may encourage publication of new data, helps data
publishers understand data consumers needs better and, when feedback
is made publicly available, enhances the consumers' collaborative
experience.
Intended Outcome
It should be possible for data consumers to provide feedback and
rate data in both human and machine-readable formats. The feedback
should be Web accessible and it should provide a URL reference to
the corresponding dataset.
Possible Approach to Implementation
Provide data consumers with one or more feedback mechanisms
including, but not limited to: a registration form, contact form,
point and click data quality rating buttons, or a comment box for
blogging.
Collect feedback in machine-readable formats to represent the
feedback and use a vocabulary to capture the semantics of the
feedback information.
How to Test
Demonstrate how feedback can be collected from data consumers.
Verify that the feedback is persistently stored. If the feedback
is made publicly available verify that a URL links back to the
published data being referenced.
Check that the feedback format conforms to a known
machine-readable format specification in current use among
anticipated data users.
Feedback should be available for both human users and computer applications.
Why
Making feedback about datasets and distributions publicly available allows users to become aware of other data consumers, supports a collaborative environment, and allows user community experiences, concerns or questions are currently being addressed. Providing feedback in a machine-readable format allows computer applications to automatically collect and process feedback about datasets.
Intended Outcome
It should be possible for humans to have access to feedback on a dataset or distribution given by one or more data consumers.
It should be possible for machines to automatically process feedback about a dataset or distribution.
Possible Approach to Implementation
Feedback can be availabe as part of an HTML Web page, but it can also be provided in a machine-readable format according to the vocabulary to describe dataset usage [[DUV]].
How to Test
Check if a human consumer can access the feedback about the dataset or distribution and check if a computer application can automatically process the feedback.
Data enrichment refers to a set of processes that can be used to
enhance, refine or otherwise improve raw or previously processed data.
This idea and other similar concepts contribute to making data a
valuable asset for almost any modern business or enterprise.
This section provides some advice to be followed by data publishers in
order to enrich data.
Enrich data by generating
new data
Enrich your data by generating new data from the raw data when doing so will enhance its value.
Why
Enrichment can greatly enhance processability, particularly for unstructured data. Missing values can be filled in, and new attributes and measures can be added. Publishing more complete datasets enhances trust. Deriving additional values that are of general utility saves users time and encourages more kinds of reuse. There are many intelligent techniques that can be used to enrich data, making the dataset an even more valuable asset.
Intended Outcome
A dataset that has missing values should be enhanced if possible to fill in those values. Additional relevant measures or attributes should be added if they enhance utility. Unstructured data can be given structure in this way as well.
Because inference-based enrichment may introduce errors into the data, values generated by such techniques should be labeled as such, and it should be possible to retrieve any original values replaced by enrichment.
Whenever licensing permits, the code used to enrich the data should be made available along with the dataset. Sharing such code is particularly important for scientific data.
Possible Approaches to Implementation
Machine learning can be readily applied to the enrichment of data. Methods include those focused on data categorization, disambiguation, entity recognition, sentiment analysis, and topification, among others. After new data is extracted, it can be provided as part of any open data format.
New data values may be derivable as simply as performing a mathematical calculation across existing columns. Other examples include visual inspection to identify features in spatial data and cross-reference to external databases for demographic information.
How to test
Look for missing values in the dataset or additional fields likely to be needed by others. Check that any data added by inferential enrichment techniques is identified as such and that any replaced data is still available. Check that code used to enrich the data is available. Check whether the metadata being extracted is in accordance with human knowledge and readable by humans.
Enrich data by also presenting it in complementary, immediately informative ways, such as visualizations, tables, Web applications, or summaries.
Why
Data published online is meant to inform others about its subject. But only posting datasets for download or API access puts the burden on consumers to interpret it. The Web offers unparalleled opportunities for presenting data in ways that let users learn and explore without having to create their own tools.
Intended Outcome
Besides making datasets available for download, processing, and reuse, publishers should give human consumers immediate insight into the data by presenting it in ways that are readily understood. Data consumers should not have to create their own tools to understand the meaning of the data.
Possible Approaches to Implementation
One very simple way to provide immediate insight is to publish an analytical summary in an HTML page. Including summative data in graphs or tables can help users scan the summary and quickly understand the meaning of the data.
If you have the means to create interactive visualizations or Web applications that use the data, you can give consumers of your data greater ability to understand it and discover patterns in it. These approaches also demonstrate its suitability for processing and encourage reuse.
How to test
Check that the dataset is accompanied by some additional interpretive content that can be perceived without downloading the data or invoking an API.
We're keen for comments about the title of this section. Which one is more suitable: Data Usage or Data Reuse?
Reusing data is another way of publishing data. It can take the form of combining existing data with other datasets, creating Web applications or visualizations, or repackaging the data in a new form, such as a translation. Data reusers have some responsibilities that are unique to that form of publishing on the Web. This section provides advice to be followed when reusing data.
Provide Feedback to the Original Publisher
When using data published by others, let them know that you are reusing their data. If you find an error or have suggestions or compliments, let them know.
Why
Publishers generally want to know whether the data they publish has been useful. Moreover, they may be required to report usage statistics in order to allocate resources to data publishing activities. Reporting your usage helps them justify putting effort toward data releases. Providing feedback repays the publishers for their efforts by directly helping them to improve their dataset for future users.
Intended Outcome
Better communication will make it easier for original publishers to determine how the data they post is being used, which in turn helps them justify publishing the data. Publishers will also be made aware of steps they can take to improve their data. This leads to more and better data for everyone.
Possible Approach to Implementation
When you begin using a dataset in a new product, make a note of the publisher’s contact information, the URI of the dataset you used, and the date on which you contacted them. This can be done in comments within your code where the dataset is used. Follow the publisher’s preferred route to provide feedback. If they do not provide a route, look for contact information for the Web site hosting the data.
How to test
Check that you have a record of at least one communication informing the publisher of your use of the data.
Find and follow the licensing requirements from the original publisher of the dataset.
Why
Licensing provides a legal framework for using someone else’s work product. By adhering to the original publisher’s requirements, you keep the relationship between yourself and the publisher friendly. You don’t need to worry about legal action from the original publisher if you are following their wishes. Understanding the initial license will help you determine what license to select for your reuse.
Intended Outcome
Data publishers will be able to trust that their work is being reused in accordance with their licensing requirements, which will make them more likely to continue to publish data. Reusers of data will themselves be able to properly license their derivative works.
Possible Approach to Implementation
Read the original license and adhere to its requirements. If the license calls for specific licensing of derivative works, choose your license to be compatible with that requirement. If no license is given, contact the original publisher and ask what the license is.
How to test
Read through the original license and check that your use of the data does not violate any of the terms.
Indicate the source of your data in the metadata for your reuse of the data. If you provide a user interface, include the citation visibly in the interface.
Why
Data is only useful when it is trustworthy. Identifying the source is a major indicator of trustworthiness in two ways: first, the user can judge the trustworthiness of the data from the reputation of the source, and second, citing the source suggests that you yourself are trustworthy as a republisher. In addition to informing the end user, citing helps publishers by crediting their work. Publishers who make data available on the Web deserve acknowledgment and are more likely to continue to share data if they find they are credited. Citation also maintains provenance and helps still others to work with the data.
Intended Outcome
End users should be able to assess the trustworthiness of the data they see, and original publishers should be recognized for their efforts. The chain of provenance for data on the Web should be traceable back to its original publisher.
Possible Approach to Implementation
You can use the Dataset Usage Vocabulary to cite the original publication of the data in metadata.
How to test
Check that the original source of any reused data is cited in the metadata provided. Check that a human-readable citation is readily visible in any user interface.
A dataset is defined as a collection of data, published or curated
by a single agent, and available for access or download in one or
more formats. A dataset does not have to be available as a
downloadable file.
A Citation may be either direct and explicit (as in the reference
list of a journal article), indirect (e.g. a citation to a more
recent paper by the same research group on the same topic), or
implicit (e.g. as in artistic quotations or parodies, or in cases of
plagiarism).
For the purposes of this WG, a Data Consumer is a person or group
accessing, using, and potentially performing post-processing steps
on data.
From: Strong, Diane M., Yang W. Lee, and Richard Y. Wang. "Data
quality in context." Communications of the ACM 40.5 (1997): 103-110.
Data format
Data Format defined as a specific convention for data
representation i.e. the way that information is encoded and stored
for use in a computer system, possibly constrained by a formal data
type or set of standards."
Data Producer is a person or group responsible for generating and
maintaining data.
From: Strong, Diane M., Yang W. Lee, and Richard Y. Wang. "Data
quality in context." Communications of the ACM 40.5 (1997): 103-110.
Data representation
Data representation is any convention for the arrangement of
symbols in such a way as to enable information to be encoded by a
data producer and later decoded by data consumers.">Data
representation
A distribution represents a specific available form of a dataset.
Each dataset might be available in different forms, these forms
might represent different formats of the dataset or different
endpoints. Examples of distributions include a downloadable CSV
file, an API or an RSS feed
A feedback forum is used to collect messages posted by consumers
about a particular topic. Messages can include replies to other
consumers. Datetime stamps are associated with each message and the
messages can be associated with a person or submitted anonymously.
To better understand why an annotation [[Annotation-Model]] was created, a
SKOS Concept Scheme is used to
show inter-related annotations between communities with more
meaningful distinctions than a simple class/subclass tree.
Data preservation
Data Preservation is defined by APA
as "The processes and operations in ensuring the technical and
intellectual survival of objects through time". This is part of a
data management plan focusing
on preservation planning and meta-data. Whether it is
worthwhile to put effort into preservation depends on the (future)
value of the data, the resources available and the opinion of the
stakeholders (= designated community).
Data archiving
Data Archiving is the set of practices around the storage and
monitoring of the state of digital material over the years.
These tasks are the responsibility of a Trusted Digital Repository
(TDR), also sometimes referred to as Long-Term
Archive Service (LTA). Often such services follow the Open
Archival Information System which defines the archival process
in terms of ingest, monitoring and reuse of data.
Data Provenance
Provenance originates from the French term "provenir" (to come
from), which is used to describe the curation process of artwork as
art is passed from owner to owner. Data provenance, in a similar
way, is metadata that allows data providers to pass details about
the data history to data users.
Data Quality
Data quality is commonly defined as “fitness for use” for a
specific application or use case.
File format
File Format is a standard way that information is encoded for
storage in a computer file. It specifies how bits are used to encode
information in a digital storage medium. File formats may be either
proprietary or free and may be either unpublished or open.
A locale is a set of parameters that defines specific data
aspects, such as language and formatting used for numeric values and
dates.
Machine Readable Data
Machine Readable Data are data formats that may be readily parsed
by computer programs without access to proprietary libraries. For
example CSV
and RDF
turtle family for graphs are machine readable, but PDF
and JPEG are not.
Sensitive data is any designated data or metadata that is used in
limited ways and/or intended for limited audiences. Sensitive data
may include personal data, corporate or government data, and
mishandling of published sensitive data may lead to damages to
individuals or organizations.
Vocabulary
Vocabulary is A collection of "terms" for a particular purpose.
Vocabularies can range from simple such as the widely used RDF
Schema, Foaf and Dublin
Core Metadata Element Set to complex vocabularies with
thousands of terms, such as those used in healthcare to describe
symptoms, diseases and treatments. Vocabularies play a very
important role in Linked Data, specifically to help with data
integration. The use of this term overlaps with Ontology.
Structured Data refers to data that conforms to a fixed schema.
Relational databases and spreadsheets are examples of structured
data.
Data on the Web Challenges
The following diagram summarizes some of the main
challenges faced when publishing or consuming data on the Web. These
challenges were identified from the DWBP
Use Cases and Requirements [[UCR]] and, as presented in the diagram, is
addressed by one or more best practices.
Best Practices Benefits
The list below describes the main benefits of applying the DWBP. Each benefit represents an improvement in the way how datasets are available on the Web.
Comprehension: humans will have a better understanding about the
data structure, the data meaning, the metadata and the nature of the
dataset.
Processability: machines will be able to automatically process and
manipulate the data within a dataset.
Discoverability machines will be able to automatically discover a
dataset or data within a dataset.
Reuse: the chances of dataset reuse by different groups of data
consumers will increase.
Trust: the confidence that consumers have in the dataset will
improve.
Linkability: it will be possible to create links between data
resources (datasets and data items).
Access: humans and machines will be able to access up to date data
in a variety of forms.
Interoperability: it will be easier to reach consensus among data
publishers and consumers.
The following table relates Best Practices and Benefits.
Best Practices and Benefits
Best Practice
Benefits
The figure below shows the benefits that data publishers will gain
with adoption of the best practices.
Feedback about the association between BP and benefits is welcome!
Use Cases Requirements x Best Practices
Requirements x Best Practices
Requirement
Best Practices
Acknowledgements
The editors gratefully acknowledge the contributions made to this
document by all members of the working group and the chairs: Hadley
Beeman, Steve Adler, Yaso Córdova, Deirdre Lee.
A new section to explain the general context and some of the main concepts used in the BP document.
A new section to present the main benefits of adopting the BP as well as the association between BP and benefits. The list of benefits and BP is auto-generated.
Inclusion of a new section on the BP Template to list the benefits of the BP. Icons were created to represent each one of the benefits.
Inclusion of new examples and the updating of some of the existing examples.
Inclusion of a glossary to explain some of the terms used in the BP document.
Inclusion of a table to show the BP associated with each one of the Use Cases requirements.
Inclusion of a table to show the benefits associated with each one of the BP.
