AW: Data Quality Vocabulary - Multiple/Derived values of a metric and Levels of granularity for dimensions and categories

Dear Antoine, all,

thanks for your response, and also apologies for my late answer.

I think the proposed solution of keeping metrics separate and using a set of properties to express their relations makes sense. 

There may be some useful properties in http://purl.org/net/QualityModel#, but for the reasons you used to argue against a fixed aggregation of values, the distinction between BaseMetrics and DerivedMetrics in this model goes probably too far, as it again forces the use of a specific set of metrics. Using properties such as dependsOn between two metrics would allow to derive a base - derived relation if used this way, but would not enforce it. And for facilitating machine processing of the results, defining further subproperties (e.g., for common statistical variables) would be useful.

Best regards,
Werner

-----Ursprüngliche Nachricht-----
Von: Antoine Isaac [mailto:aisaac@few.vu.nl] 
Gesendet: Sonntag, 06. März 2016 22:09
An: Bailer, Werner; Debattista, Jeremy
Cc: public-dwbp-comments@w3.org; Russegger, Silvia; Orgel, Thomas; Höffernig, Martin; Ehgarter, Stephan
Betreff: Re: Data Quality Vocabulary - Multiple/Derived values of a metric and Levels of granularity for dimensions and categories

Dear Werner,

We are sorry not to have come back to you earlier. We had a lot of work, and couldn't focus as much on continuing the discussion....
Now the document has matured - you can see the latest version at http://w3c.github.io/dwbp/vocab-dqg.html - we can come back with some elements of answer. Starting with two of your issues at once, as they seemed quite related.


> Thanks for creating the issues related to different aspects of the comments, I'm responding to Jeremy's answers for each of them.
>
>> https://www.w3.org/2013/dwbp/track/issues/222 - Multiple/Derived 
>> values of a metric " In theory, there is only one value for a metric - others are derivatives."
>
> I agree -  "in theory". There are cases where an output value can indeed be derived (e.g., normalised vs absolute value). And one could always split metrics so that they only produce a single value (in fact, that's what we did now as I wrote in my mail).
>
> However, in my understanding daq:Observation is meant to represent the result of evaluating a metric at a specific time on a specific dataset. I do not see how it could be used to express e.g.  different statistical parameters of the same metric. For example, expressing the mean and max of non-empty fields in a dataset of are two values collected based on the same raw values, but one metric cannot be directly derived from the others. They could be separate metrics, but is then it would make sense group them.
>

We agree with you: the individual metrics shouldn't stay isolated!

Extending on Jeremy's point, however, it seems dangerous for interoperability purposes to create metrics that lump different values.
Especially, different applications could judge that different aggregates are relevant for a same base measure. Let's say that for a base metric 'number of outbound links for a resource', an application prefers to compute the average, another prefers the min and max. If we split the base measurement from the aggregates, then we still have something easily comparable and re-usable across applications: the number of outbound link, as one value of a metric. If we lump them together, then applications would have to figure out what is comparable and re-usable in the 'composed' Measurement. This is doable, but could require an apparatus of specializations of the dqv:value property and relation between the values of it, maybe even reification. This would be even more complex than splitting Metrics and Measurement and relating them.


The solution we're currently envisioning, is based on a specific understanding of your suggestion "They could be separate metrics, but is then it would make sense group them.".
It's being discussed at
https://lists.w3.org/Archives/Public/public-dwbp-wg/2016Feb/0045.html

Essentially, we're looking into using a small set of properties to indicate how Metrics are based on one another.
Here is an external vocabulary proposal that includes some properties, like dependsOn or usedToObtain:
http://purl.org/net/QualityModel#

We've not discussed the matter enough now to be sure that in the end we would re-use this vocabulary or another.
But at this stage we think we have waited enough and prefer to ask your opinion now!

Thanks,

Antoine



>
> -----Ursprüngliche Nachricht-----
> Von: Antoine Isaac [mailto:aisaac@few.vu.nl]
> Gesendet: Sonntag, 06. Dezember 2015 17:40
> An: Debattista, Jeremy; Bailer, Werner
> Cc: public-dwbp-comments@w3.org; Russegger, Silvia; Orgel, Thomas; 
> Höffernig, Martin; Ehgarter, Stephan
> Betreff: Re: Data Quality Vocabulary - feedback welcome!
>
> Dear Werner,
>
> As one of editor of the DQV spec, I'd like to thank you once again for the relevant feedback. This is priceless for us! Especially as you are actually looking at using DQV...
>
> We had not reacted after Jeremy's answer below, as it seemed a great way to get the discussion going. But we are nearing the publication of a new DQV draft, and we would really like that commenters are happy enough with the way we tackle their input.
>
> So, taking you to your word when you say you "are happy to continue 
> the discussion and provide feedback on future iterations" :-)
>
> We have raised issues to represent the different part of your comments:
> https://www.w3.org/2013/dwbp/track/issues/222 - Multiple/Derived 
> values of a metric
> https://www.w3.org/2013/dwbp/track/issues/223 - Parameters for metrics
> https://www.w3.org/2013/dwbp/track/issues/224 - Expected Data type for 
> metrics
> https://www.w3.org/2013/dwbp/track/issues/225 - Levels of granularity 
> for dimensions and categories
>
> We hope these capture your concerns, and will mail you about them in different emails.
> In the meantime we will also mention these issues in the current editor's draft of DQV [1].
>
> Best,
>
> Antoine
>
> [1] http://w3c.github.io/dwbp/vocab-dqg.html

>
>
>>> There are however a few observation from the use of DQV that we'd like to share with the group.
>>>
>>> 1. daq:metric
>>> a. multiple values of one metric
>>>
>>> We found that there are metrics where a single output value may not be sufficient. In particular, this applies to statistics (which are listed as one dimension in the draft spec). For example, one may want to express the mean, min or max of a metric over a dataset, or providing an absolute and a relative (normalized) value for the same metric. Of course this could be done by defining multiple metrics, but then one would need a mechanism to group/link them or express their dependency.
>>>
>>> In the EBU, the working group on quality control [2] has defined a data model for the somewhat related problem of describing quality of audiovisual content (with XML serialisations so far, not RDF). This model supports multiple output values, that can be typed. For DQV, this could for example be achieved by having multiple values, and defining subproperties of daq:value.
>>
>> In theory, there is only one value for a metric - others are derivatives. With a daq:Observation, such derivatives should be easily defined by creating external data cube measure property.
>>
>>> b. parameters
>>>
>>> For some metrics, input parameters could be required. E.g., there have been recent publications on metadata quality which use weights or target values in the metrics. For descriptions with quality measurements that are self-contained, it would be required to include the values of such parameters in the description of the metric.
>>
>> A daq:Metric (which is the equivalent class of dqv:Metric) has the property daq:requires. The purpose of that property is exactly for input parameters.
>>
>>
>>> c. daq:expectedDataType
>>>
>>> This property from DAQ is defined to have range xsd:anySimpleType. While it seems useful to define the expected data type for a metric, a simple type may too narrow: in many cases a metric will be determined on a data record or a subgraph.
>>
>> It will be taken into consideration - although I’m not sure it works well with data cube. Please can you provide us (or me) with an example where a quality metric returns a data record or sub graph?
>>
>>>
>>> 2. Dimensions and categories
>>>
>>> The dimensions proposed seem quite high-level, so it is difficult to think of categories that are more general and group dimensions. In contrast, it seems in some cases desirable to have a level between dimensions and metrics. For example, we are dealing with assessing mapping quality. The metrics fall in the dimension of accuracy (i.e., does the output of the mapping process represent the object less accurately), and form a specific group there. To make the distinction of the different levels more confusing, the note in 7.3 Processability currently says "Level on the 5-star scale", which sounds more like a metric than a dimension (there could of course be metrics aggregating results from other metric, daq:requires could be used to express such a dependency).
>>> We are not sure if there is a strong need for categories, we would rather propose to consider nesting multiple levels of dimensions to allow grouping.
>>
>> I’m not sure if I understood “nesting multiple levels of dimensions” correctly, but a category groups a set of dimensions which have a common type of information as a quality indicator. For example the Accessibility category groups dimensions such as Availability, Security and Performance. Each of these dimensions have a number of different metrics, each assessing different aspect of a dimension. This is how we define Category-Dimension-Metric in daq:
>>
>>      /A *Quality Dimension* is a characteristic of a dataset relevant to the consumer (e.g. Availability of a dataset)./
>>      /
>>      /
>>      /A *Quality Metric* is concrete quality measure for a concrete quality indicator usually associ- ated with a measuring procedure. This assessment procedure returns a score, which we also call the value of the metric. There are usually multi- ple metrics per dimension; e.g., availability can be measured by the accessibility of a SPARQL endpoint, or of an RDF dump. The value of a metric can be numeric (e.g., for the metric “human-readable labelling of classes, properties and entities”, the percentage of entities having an rdfs:label or rdfs:comment) or boolean (e.g. whether or not a SPARQL endpoint is accessible)./
>>      /
>>      /
>>      /A *Category* is a group of quality dimensions in which a common 
>> type of information is used as quality indicator (e.g. Accessibility, 
>> which comprises not only availability but also dimensions such as 
>> security or performance). Grouping the dimensions into categories 
>> helps to organise the space of all quality aspects, given their large 
>> number./
>>
>>
>>
>> I hope this helps.
>>
>> Best Regards,
>> Jeremy
>>

Received on Tuesday, 19 April 2016 15:58:23 UTC