Re: Subsetting data

Eric,
My use case is not so much when a researcher wants to treat a subset of a larger dataset as its own dataset. Rather, I'm thinking of something like the OpenMSI use case, where interactivity with a very large dataset is made possible by moving smallish chunks of the data from server to client only as needed. This is the concept of lazy loading. There is no treatment of the subsets as datasets in their own right. They are cuts or lines through a data cube, returned as a response to a s is a common use case at NERSC.
-Annette

Sent from a keyboard-challenged device

> On Jan 4, 2016, at 8:56 AM, Eric Stephan <ericphb@gmail.com> wrote:
> 
> Hi all and Happy New Year! 
> 
> This is an incredibly helpful discussion on data subset identification both from a data on the web perspective as well as citing data on the web.  I'm really behind on catching up with email after the holidays and will need to play catch up in this discussion.  
> 
> Frans I like this overall points you are covering.
> 
> Annette, I agree with the idea that at least in the physical sciences that I'm supporting if a scientist finds an interesting data subset that they will extract this from the original dataset more often than not, but lack the means of showing the association between the original dataset and the subset.  Describing the extraction process seems provenance oriented and PROV-O could be used to describe the entities and activities involved to subset the data.
> 
> I do like the idea where data subset identification can be correlated to queries where data is extracted live.  I'm not familiar with opensearch, but I'm imagining this would be helpful in situations where communities that share data on the web and heavily use domain APIs based on their particular formats.
> 
> Eric Stephan
> Pacific Northwest National Labs  
> 
> 
>> On Mon, Jan 4, 2016 at 4:48 AM, Frans Knibbe <frans.knibbe@geodan.nl> wrote:
>> 
>> 
>> 
>> 2016-01-04 13:20 GMT+01:00 Peter Baumann <p.baumann@jacobs-university.de>:
>>> Hi Frans,
>>> 
>>> data partitioning is an implementation detail which serves to quicker determine the subset.
>> 
>> But that does not need to be the only reason for  dataset partitioning. I can think of some others:
>> It could allow complete automatic retrieval of a complete dataset without the need for a data dump (in a specific format) and without the need for a specialized API. That could be very important for building remote indexes (e.g. by a search engine);
>> It allows automatic representation of the data in a human friendly way (small enough HTML pages with annotation and navigation);
>> It allows caching of meaningful and self-describing chunks of data.
>>  
>>> As such, any subsetting or querying interface should remain agnostic of it, otherwise it runs the risk of supporting a particular implementation (and there are quite a few specialized implementations out there).
>> 
>> I think the concept of dataset partitioning could be handled in a very general way. Perhaps the only required elements are:
>> express that something (a web resource) is a dataset (and could therefore be a subset or a superset);
>> express that a dataset is a superset or subset of another dataset. 
>> This could work with any type of data and with any query interface, I think.
>> 
>> Regards,
>> Frans
>>> 
>>> 
>>> -Peter
>>> 
>>> 
>>> 
>>>> On 2016-01-04 13:14, Frans Knibbe wrote:
>>>> 
>>>> 
>>>> 2016-01-01 10:33 GMT+01:00 Phil Archer <phila@w3.org>:
>>>>> 
>>>>> 
>>>>>> On 31/12/2015 10:54, Frans Knibbe wrote:
>>>>>> Phil,
>>>>>> 
>>>>>> Thank you for bringing up an interesting subject at a time where not much
>>>>>> seems to be going on.
>>>>>> 
>>>>>> I think a key question is: Which data should be returned when a dataset URI
>>>>>> is dereferenced?
>>>>>> 
>>>>>> And I think the answer should be: at least the metadata describing the
>>>>>> dataset or the subset, and optionally the actual data.
>>>>> 
>>>>> I'd say: If I ask for the current temperature in Amsterdam, that's what I want. Good practice would be to include metadata, or links to it (dcterms:isPartOf <allTemperaturesInNL>).
>>>>> 
>>>>> I don't disagree that those things are important, metadata clearly is  - and goodness knows I like links :-) It's a scoping/capacity to deliver question.
>>>> 
>>>> It probably also has to do with the kind of subsets we have in mind. Perhaps two kinds can be distinguished:
>>>> 
>>>> 1) subsets that are predetermined subdivisions of a dataset. The dataset partioning is done by the data publisher and likely to be documented (e.g. by means of dataset metadata). The partioning is likely to be obvious and useful to many consumers, e.g. time slices in a temporal data set, tiles or administrative subdivisions in a geographic data set.
>>>> 2) subsets that are the result of an ad hoc query, ephemeral subsets. 
>>>> 
>>>> The two do not need to be mutually exclusive. 
>>>> 
>>>> I particularly like the possibilities of agreed best practices for the first type of subsets. A predetermined partioning of a dataset could result in at least one logical tree of data, in which branches and leaves are well linked and thus allow for easy navigation within the dataset, for humans and machines (crawlers) alike. I think there could be interesting possibilities for common practices in structuring and describing a partioned dataset, a main advantage being improved discoverability because a crawler could easily access all the data in a dataset from the root node, or from any node in a dataset. For humans the predetermined structure should be useful too because it could help serving HTML pages that are not to big, are well documented and are easy to navigate.
>>>> 
>>>> How do you see the relationship between the topic of persistent identifiers for subsets and the SDWWG requirements for linkability, discoverability and crawlability of data?
>>>> 
>>>> Regards,
>>>> Frans
>>>> 
>>>> 
>>>>  
>>>>> 
>>>>> Phil
>>>>> 
>>>>>> 
>>>>>> When discussing datasets and subsets it is good to look at the Vocabulary
>>>>>> of Interlinked Datasets (VoID) <http://www.w3.org/TR/void/>, although its
>>>>>> scope could be too narrow because it is intended to be used for RDF data.
>>>>>> It can be used to make clear that a chunk of data describes a dataset (
>>>>>> void:Dataset <http://rdfs.org/ns/void#Dataset>) and has subsets (void:subset
>>>>>> <http://www.w3.org/TR/void/#subset>). The Data Catalog Vocabulary
>>>>>> <http://www.w3.org/TR/vocab-dcat/> has a broader scope (it can be used for
>>>>>> any dataset) and has its own definition of a dataset (dcat:Dataset
>>>>>> <http://www.w3.org/ns/dcat#Dataset>). DCAT does not seem to have a way of
>>>>>> identifying subsets, but I guess dcterms:hasPart
>>>>>> <http://purl.org/dc/terms/hasPart> and dcterms:isPartOf
>>>>>> <http://purl.org/dc/terms/isPartOf> can be used to express parent-child
>>>>>> relationships between data collections (dataset mereology).
>>>>>> 
>>>>>> So let's assume it is possible to indicate that a set of data describe a
>>>>>> dataset and that it is possible to express in a general way that the
>>>>>> dataset is a subset of a parent dataset and itself is the parent of a
>>>>>> collection of subsets. The data that are returned when a dataset URI is
>>>>>> dereferenced could then include:
>>>>>> 
>>>>>> 
>>>>>>     - A link to the parent dataset (if there is one)
>>>>>>     - Links to child datasets (if they exist)
>>>>>>     - Descriptions of how to get the actual data (if there are not included
>>>>>>     in the response), for example the URI of a SPARQL endpoint or the URIs of
>>>>>>     other standard web APIs
>>>>>>     - Other general metadata, like spatial extent, temporal extent, human
>>>>>>     readable labels, subject(s), etc.
>>>>>>     - The actual data that from the dataset
>>>>>> 
>>>>>> A recommendation or good practice could be to include the actual data OR
>>>>>> point to subsets. That way there is never a dead end when links are
>>>>>> followed. A data provider could decide the best level of a subset returning
>>>>>> actual data, for example when the amount of data is manageable.
>>>>>> 
>>>>>> What I particularly like about this approach is that if the data server
>>>>>> supports HTML (or another format that is supported by web crawlers), we
>>>>>> will have satisfied the crawlability requirement
>>>>>> <http://www.w3.org/TR/sdw-ucr/#Crawlability> and the discoverability
>>>>>> requirement <http://www.w3.org/TR/sdw-ucr/#Discoverability>.  A web crawler
>>>>>> 
>>>>>> could use any dataset URI as a starting point and by recursively visiting
>>>>>> all links always have access to the complete dataset, in a way that does
>>>>>> not require any fancy querying. I hope the search engine people (Ed,
>>>>>> Charles) can confirm this...
>>>>>> 
>>>>>> Another thing I like about this approach is that the spatial properties of
>>>>>> a dataset can be helpful in partioning a dataset into managable subsets. An
>>>>>> obvious method would be to use administrative (mereological) relationship:
>>>>>> A European dataset has a subsets for each country, a country dataset has
>>>>>> subsets for each province, and so on. If that possibility is absent it
>>>>>> should always be possible to use a tiling mechanism to partition the
>>>>>> dataset into subsets. I like to think of this as a nice example of how
>>>>>> geospatial practice can be benificial to the Web as a whole.
>>>>>> 
>>>>>> By the way, I would like to look at the transport.data.gov.uk examples, but
>>>>>> I get 404s.
>>>>>> 
>>>>>> Regards,
>>>>>> Frans
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 2015-12-30 19:31 GMT+01:00 Phil Archer <phila@w3.org>:
>>>>>> 
>>>>>>> At various times in recent months I have promised to look into the topic
>>>>>>> of persistent identifiers for subsets of data. This came up at the SDW F2F
>>>>>>> in Sapporo but has also been raised by Annette in DWBP. In between festive
>>>>>>> activities I've been giving this some thought and have tried to begin to
>>>>>>> commit some ideas to a page [1].
>>>>>>> 
>>>>>>> During the CEO-LD meeting, Jeremy pointed to OpenSearch as a possible way
>>>>>>> forward, including its geo-temporal extensions defined by the OGC. There is
>>>>>>> also the Linked Data API as a means of doing this, and what they both have
>>>>>>> in common is that they offer an intermediate layer that turns a URL into a
>>>>>>> query.
>>>>>>> 
>>>>>>> How do you define a persistent identifier for a subset of a dataset? IMO
>>>>>>> you mint a URI and say "this identifies a subset of a dataset" - and then
>>>>>>> provide a means of programmatically going from the URI to a query that
>>>>>>> returns the subset. As long as you can replace the intermediate layer with
>>>>>>> another one that also returns the same subset, we're done.
>>>>>>> 
>>>>>>> The UK Government Linked Data examples tend to be along the lines of:
>>>>>>> 
>>>>>>> http://transport.data.gov.uk/id/stations
>>>>>>> returns a list of all stations in Britain.
>>>>>>> 
>>>>>>> http://transport.data.gov.uk/id/stations/Manchester
>>>>>>> returns a list of stations in Manchester
>>>>>>> 
>>>>>>> http://transport.data.gov.uk/id/stations/Manchester/Piccadilly
>>>>>>> identifies Manchester Piccadilly station.
>>>>>>> 
>>>>>>> All of that data of course comes from a single dataset.
>>>>>>> 
>>>>>>> Does this work in the real worlds of meteorology and UBL/PNNL?
>>>>>>> 
>>>>>>> Phil.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> [1] https://github.com/w3c/sdw/blob/gh-pages/subsetting/index.md
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> 
>>>>>>> 
>>>>>>> Phil Archer
>>>>>>> W3C Data Activity Lead
>>>>>>> http://www.w3.org/2013/data/
>>>>>>> 
>>>>>>> http://philarcher.org
>>>>>>> +44 (0)7887 767755
>>>>>>> @philarcher1
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> -- 
>>>>> 
>>>>> 
>>>>> Phil Archer
>>>>> W3C Data Activity Lead
>>>>> http://www.w3.org/2013/data/
>>>>> 
>>>>> http://philarcher.org
>>>>> +44 (0)7887 767755
>>>>> @philarcher1
>>>> 
>>> 
>>> -- 
>>> Dr. Peter Baumann
>>>  - Professor of Computer Science, Jacobs University Bremen
>>>    www.faculty.jacobs-university.de/pbaumann
>>>    mail: p.baumann@jacobs-university.de
>>>    tel: +49-421-200-3178, fax: +49-421-200-493178
>>>  - Executive Director, rasdaman GmbH Bremen (HRB 26793)
>>>    www.rasdaman.com, mail: baumann@rasdaman.com
>>>    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
>>> "Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)
>>> 
>>> 
>> 
>