Re: 15 Ways to Think About Data Quality (Just for a Start)

On Fri, 2011-04-08 at 21:10 -0400, glenn mcdonald wrote:
> I don't think data quality is an amorphous, aesthetic, hopelessly
> subjective topic. Data "beauty" might be subjective, and the same data
> may have different applicability to different tasks, but there are a
> lot of obvious and straightforward ways of thinking about the quality
> of a dataset independent of the particular preferences of individual
> beholders. Here are just some of them:
> 
> 
> 1. Accuracy: Are the individual nodes that refer to factual
> information factually and lexically correct. Like, is Chicago spelled
> "Chigaco" or does the dataset say its population is 2.7?
> 
> 
> 2. Intelligibility: Are there human-readable labels on things, so you
> can tell what a thing is when you're looking at? Is there a model, so
> you can tell what questions you can ask? If a thing has multiple
> labels (or a set of owl:sameAs things havemlutiple labels), do you
> know which (or if) one is canonical?
> 
> 
> 3. Referential correspondence: If a set of data points represents some
> set of real-world referents, is there one and only one point per
> referent? If you have 9,780 data points representing cities, but 5 of
> them are "Chicago", "Chicago, IL", "Metro Chicago", "Metropolitain
> Chicago, Illinois" and "Chicagoland", that's bad.
> 
> 
> 4. Completeness: Where you have data representing a clear finite set
> of referents, do you have them all? All the countries, all the states,
> all the NHL teams, etc? And if you have things related to these sets,
> are those projections complete? Populations of every country?
> Addresses of arenas of all the hockey teams?
> 
> 
> 5. Boundedness: Where you have data representing a clear finite set of
> referents, is it unpolluted by other things? E.g., can you get a list
> of current real countries, not mixed with former states or fictional
> empires or adminstrative subdivisions?
> 
> 
> 6. Typing: Do you really have properly typed nodes for things, or do
> you just have literals? The first president of the US was not "George
> Washington"^^xsd:string, it was a person whose name-renderings include
> "George Washington". Your ability to ask questions will be constrained
> or crippled if your data doesn't know the difference.
> 
> 
> 7. Modeling correctness: Is the logical structure of the data properly
> represented? Graphs are relational databases without the crutch of
> "rows"; if you screw up the modeling, your queries will produce
> garbage.
> 
> 
> 8. Modeling granularity: Did you capture enough of the data to
> actually make use of it. ":us :president :george_washington" isn't
> exactly wrong, but it's pretty limiting. Model presidencies, with
> their dates, and you've got much more powerful data.
> 
> 
> 9. Connectedness: If you're bringing together datasets that used to be
> separate, are the join points represented properly. Is the US from
> your country list the same as (or owl:sameAs) the US from your list of
> presidencies and the US from your list of world cities and their
> populations?
> 
> 
> 10. Isomorphism: If you're bring together datasets that used to be
> separate, are their models reconciled? Does an album contain songs, or
> does it contain tracks which are publications of recordings of songs,
> or something else? If each data point answers this question
> differently, even simple-seeming queries may be intractable.
> 
> 
> 11. Currency: Is the data up-to-date?
> 
> 
> 12. Directionality: Can you navigate the logical binary relationships
> in either direction? Can you get from a country to its presidencies to
> their presidents, or do you have to know to only ask about presidents'
> presidencies' countries? Or worse, do you have to ask every question
> in permutations of directions because some data asserts things one way
> and some asserts it only the other?
> 
> 
> 13. Attribution: If your data comes from multiple sources, or in
> multiple batches, can you tell which came from where?
> 
> 
> 14. History: If your data has been edited, can you tell how and by
> whom?
> 
> 
> 15. Internal consistency: Do the populations of your counties add up
> to the populations of your states? Do the substitutes going into your
> soccer matches balance the substitutes going out?

That's a fantastic list and should be recorded on a wiki somewhere!

A minor quibble, not sure about Directionality. You can follow an RDF
link in both directions (at least in SPARQL and any RDF API I've worked
with).  I would be inclined to generalize and rephrase this as ...

"Consistency of modelling: whichever way you make modelling decisions
such as direction of relations (from country to president, from
president to country) it is done consistently so you don't have to ask
many permutations of the same query." 
Possible additions:

"Licensed: the license under which the data can be used is clearly
defined, ideally in a machine checkable way."

"Sustainable: there is some credible basis for believing the data will
be maintained as current (e.g. backed by some appropriate organization
or by a sufficiently large group of individuals, has been updated
frequently in the past)."

"Authoritative: is the provider of the data a credible authority on the
subject. For example, in the UK then Companies House has the definitive
information on registered UK companies and no amount of crowd sourcing
can change that fact that if the company is not registered with them
then it is not registered :)"

Dave

Received on Tuesday, 12 April 2011 08:22:08 UTC