Re: Exposing datasets with DCAT (partitioning, subsets..)

Hi Jon,
I agree completely here. Many time we are 'forced' to partition data due to
availability of, or improvements in query techniques... or simply requests
from customers!
Our data(set) partitioning strategies are dependent on a multitude of data
modeling assumptions and decisions. They can also be determined by the
hardware and software we are using to persist and query the data.
Lewis

On Wed, Feb 3, 2016 at 4:56 AM, Jon Blower <j.d.blower@reading.ac.uk> wrote:

> Hi all,
>
> Just to chip in - I think that dataset partitioning is *not* (necessarily)
> intrinsic to the dataset [1], but is a property of data distribution (hence
> perhaps in scope for DCAT). A dataset might be partitioned differently
> depending on user preference. Some users may prefer a geographic
> partitioning, others may prefer a temporal partitioning. Still others might
> want to partition by variable. One can imagine different catalogues serving
> the “same” data to different users in different ways (and in fact this does
> happen with large-volume geographic data like satellite imagery or global
> models).
>
>  like to think about dataset partioning as something simple, needing only
> three semantic ingredients: being able to say that a resource is a dataset,
> and being able to point to subsets and supersets.
>
>
> I agree with this. I think this is the “level zero” requirement for
> partitioning.
>
> [1] Actually, it probably depends on what you mean by "the dataset". If
> you mean the logical entity, then the partitioning is not a property of the
> dataset. But if you regard the dataset as a set of physical files then
> maybe the partitioning *is* a property of the dataset.
>
> Cheers,
> Jon
>
>
> On 3 Feb 2016, at 11:34, Maik Riechert <m.riechert@reading.ac.uk> wrote:
>
> Hi Frans,
>
> In my opinion, it all depends on how the actual data is made available. If
> it's a nice (possibly standard) API, then just link that as a distribution
> and you're done I would say. Clients can explore subsets etc. through that
> API (which in itself should be self-describing and doesn't need any further
> metadata at the Distribution level, except media type if possible).
>
> However, if you really *just* have a bunch of files, as is quite common
> and which may be ok depending on data volume and intended users, then it
> gets more complicated if you want to allow efficient machine access without
> first being forced to download everything.
>
> So, yes, partitioning is intrinsic to the dataset, and that detail is
> exposed to DCAT to allow more efficient access, both for humans and
> machines. It is an optimization in the end, but in my opinion a quite
> useful one.
>
> I wonder how many different partition strategies are really used in the
> wild.
>
> Cheers
> Maik
>
> Am 03.02.2016 um 11:13 schrieb Frans Knibbe:
>
> Hello Andrea, all,
>
> I like to think about dataset partioning as something simple, needing only
> three semantic ingredients: being able to say that a resource is a dataset,
> and being able to point to subsets and supersets. DCAT does not seem
> necessary for those three. Is there really a need to see dataset partioning
> as DCAT territory? DCAT is a vocabulary for data catalogs, I see dataset
> partioning as something intrinsic to the dataset - its structure.
>
> That said, data about the structure of a dataset is metadata so it is
> interesting to think about how data and metadata are coupled. For easy
> navigation through the structure (by either man or machine) it is probably
> best to keep the data volume small - metadata only. But it would be nice to
> have the option to get the actual data from any dataset (at any structural
> level). That means that additonial elements are needed: a indication of
> ways to get the actual data, dcat:Distribution for instance. Also an
> indication of size of the actual data would be very useful, to help decide
> to get the data or to dig a bit deeper for smaller subsets. Only at the
> highest level of the structure, the leaves of the tree, could the actual
> data be returned by default. A friendly data provider will take care that
> those subsets contain manageable volumes of data.
>
> My thoughts have little basis in practice, but I am trying to set up an
> experiment with spatially partioned data. I think there are many
> interesting possibilities. I hope to be able to share something practical
> with the group soon.
>
> Regards,
> Frans
>
>
>
>
>
> 2016-02-03 10:05 GMT+01:00 Andrea Perego <andrea.perego@jrc.ec.europa.eu>:
>
>> Many thanks for sharing this work, Maik!
>>
>> Just a couple of notes from my side:
>>
>> 1. Besides temporal coverage, it may be worth adding in your scenarios
>> also spatial coverage as another criterion of dataset partitioning.
>> Actually, both criteria are frequently used concurrently.
>>
>> 2. In many of the scenarios you describe, dataset subsets are modelled as
>> datasets. An alternative would be to model them just as distributions. So,
>> I wonder whether those scenarios have requirements that cannot be met by
>> the latter option.
>>
>> Some more words on point (2):
>>
>> As you probably know, there has been quite a long discussion in the
>> DCAT-AP WG concerning this issue. The main points are probably summarised
>> in the conversation recorded here:
>>
>>
>> https://joinup.ec.europa.eu/asset/dcat_application_profile/issue/mo12-grouping-datasets
>>
>> Of course, in DCAT-AP the objective was how to describe dataset subsets,
>> and not about criteria for dataset subsetting.
>>
>> Notably, the discussion highlighted two different approaches: (a) dataset
>> subsets modelled as datasets or (b) dataset subsets modelled simply as
>> distributions.
>>
>> I don't see the two scenarios above as mutually exclusive. You can use
>> one or the other depending of your use case and requirements. And you can
>> use both (e.g., referring to point (1): time-related subsets modelled as
>> child datasets, and their space-related subsets as distributions). However,
>> I personally favour the idea of using distributions as the recommended
>> option, and datasets only if you cannot do otherwise. In particular, I see
>> two main issues with the dataset-based approach:
>>
>> - It includes an additional step to get to the data (dataset -> dataset
>> -> distribution). Moreover, subsetting can be recursive - which increases
>> the number of steps needed to get to the data.
>>
>> - I understand that your focus is on data discovery from a machine
>> perspective. However, looking at how this will be reflected in catalogues
>> used by people, the result is that you're going to have a record for each
>> child dataset, in addition to the parent one. This scenario is quite
>> typical nowadays (I know quite a few examples of tens of records having the
>> same title, description, etc. - or just a slightly different one), and it
>> doesn't help at all people trying to find what they're looking for.
>>
>> Thanks
>>
>> Andrea
>>
>>
>>
>> On 02/02/2016 12:02, Maik Riechert wrote:
>>
>>> Hi all,
>>>
>>> There has been a lot of discussion about subsetting data. I'd like to
>>> give a slightly different perspective which is purely motivated from the
>>> point of view of someone who wants to publish data, and in parallel
>>> someone who wants to discover and access that data without much hassle.
>>>
>>> Of course it is hard to think about all scenarios, so I picked what I
>>> think are common ones:
>>> - a bunch of static data files without any API
>>> - an API without static data files
>>> - both
>>>
>>> And then some specific variations on what structure the data has (yearly
>>> data files, daily, or another dimension used as splitting point, such as
>>> spatial).
>>>
>>> It is in no way final or complete and may even be wrong, but here is
>>> what I came up with:
>>> https://github.com/ec-melodies/wp02-dcat/wiki/DCAT-partitioning-ideas
>>>
>>> So it always starts by looking at what data exists and how it is
>>> exposed, and based on those constraints I tried to model that as DCAT
>>> datasets, sometimes with subdatasets. Again, it is purely motivated from
>>> a machine-access point of view. There may be other things to consider.
>>>
>>> The point of this wiki page is to have something concrete to discuss
>>> about and not just abstract ideas. It should uncover problems, possibly
>>> solutions, perspectives... etc.
>>>
>>> Happy to hear your thoughts,
>>> Maik
>>>
>>>
>> --
>> Andrea Perego, Ph.D.
>> Scientific / Technical Project Officer
>> European Commission DG JRC
>> Institute for Environment & Sustainability
>> Unit H06 - Digital Earth & Reference Data
>> Via E. Fermi, 2749 - TP 262
>> 21027 Ispra VA, Italy
>>
>> https://ec.europa.eu/jrc/
>>
>>
>
>
>


-- 
*Lewis*