RE: hcls dataset description comments--Dataset Descriptions vs. PROV

hi all,



finally finished my initial attempt at adding a section on datasets in
workflows.  had some brief discussions with melissa and she and carole may
have some additions.  i tried to keep it short and sweet, to really get
into it could be an entire note on its own.



kim, you should see my pull request.  hope to make the call next week.



cheers,

michael



Michael Miller

Software Engineer

Institute for Systems Biology





*From:* Joachim Baran [mailto:joachim.baran@gmail.com]
*Sent:* Monday, August 11, 2014 8:00 AM
*To:* Michael Miller
*Cc:* Stian Soiland-Reyes; w3c semweb hcls
*Subject:* Re: hcls dataset description comments--Dataset Descriptions vs.
PROV



Great! Before you send the pull request, please make sure that W3C's HTML
validation passes: http://validator.w3.org/#validate_by_input



Kim



On 9 August 2014 10:15, Michael Miller <Michael.Miller@systemsbiology.org>
wrote:

hi kim,

i've made decent progress and expect to have something mid-week, if all goes
well (as a pull request, tho no guarantee on the formatting!)


cheers,
michael

Michael Miller
Software Engineer
Institute for Systems Biology

> -----Original Message-----

> From: Joachim Baran [mailto:joachim.baran@gmail.com]

> Sent: Friday, August 08, 2014 6:46 PM
> To: Michael Miller

> Cc: Stian Soiland-Reyes; w3c semweb hcls
> Subject: Re: hcls dataset description comments--Dataset Descriptions vs.
> PROV
>
> Hello,
>
>   Has there been an update to this? Preferably a pull request?
>
> Thanks,
>
> Kim
>
>
>
> > On Aug 5, 2014, at 8:13 AM, Michael Miller
> <Michael.Miller@systemsbiology.org> wrote:
> >
> > hi stian,
> >
> > thanks much, very useful!
> >
> > cheers,
> > michael
> >
> > Michael Miller
> > Software Engineer
> > Institute for Systems Biology
> >
> >
> >> -----Original Message-----
> >> From: stian@mygrid.org.uk [mailto:stian@mygrid.org.uk] On Behalf Of
> Stian
> >> Soiland-Reyes
> >> Sent: Tuesday, August 05, 2014 5:38 AM
> >> To: Michael Miller
> >> Cc: Joachim Baran; w3c semweb hcls
> >> Subject: Re: hcls dataset description comments--Dataset Descriptions
> >> vs.
> >> PROV
> >>
> >> Just some inputs:
> >>
> >>
> >> PROV defines prov:wasDerivedFrom which in broad sense describes such
> a
> >> relationset between datasets. However you do not know anything more
> >> about what kind of derivation we are talking about.
> >>
> >>
> >> In PAV we found the need to specialize three types of derivation:
> >>
> >> pav:retrievedFrom -
> >> http://purl.org/pav/html#http://purl.org/pav/retrievedFrom
> >> .. a byte-for-byte download
> >>
> >> pav:importedFrom -
> >> http://purl.org/pav/html#http://purl.org/pav/importedFrom
> >> .. a somewhat equivalent form of the source, but after some kind of
> >> transformation or selection (e.g. CSV -> XML)
> >>
> >> pav:derivedFrom -
> >> http://purl.org/pav/html#http://purl.org/pav/derivedFrom
> >> .. when the new resource has been further refined or modified
> >> (somewhat adding additional knowledge)
> >>
> >>
> >> If you are simply concatenating several dataset, then multiple
> >> pav:importedFrom statements would make sense. If further knowledge is
> >> added, say by reasoning or calculation, then pav:derivedFrom would
> >> make sense.
> >>
> >>
> >> Now if you want to detail exactly how those datasets have been
> >> combined, I think you are right that would make sense to break down
> >> the derivation using PROV statements, e.g. a series of activities,
> >> generation and usage. How to describe these activities (e.g.
> >> subclasses and properties) will be specific to each case.
> >>
> >>
> >>
> >> If the process you generated the dataset with somewhat resembles a
> >> dataflow, you might be interested in the wfprov and wfdesc ontologies
> >> that specialize PROV to define a WorkflowRun of steps of ProcessRuns,
> >> which can be related to a common workflow description (e.g. a
> >> prov:Plan):
> >>
> >> http://purl.org/wf4ever/model#wfprov
> >>
> >> OPMW is a similar approach:
> >> http://www.opmw.org/model/OPMW/
> >>
> >>
> >>
> >> On 4 August 2014 17:44, Michael Miller
> >> <Michael.Miller@systemsbiology.org> wrote:
> >>> hi all,
> >>>
> >>>
> >>>
> >>> as you are all undoubtedly aware, a major, if not the major TCGA
> >>> dataset
> >> use
> >>> cases revolve around taking the 3rd level data from the TCGA dcc
> >> repository
> >>> and doing analysis, producing 4th level data such as clusters, pca,
> >>> etc.
> >>> one of the things we do here at ISB is produce an intermediate data
> >>> step
> >>> that combines the different platforms (mRNA, miRNA, RPPA, METH,
> etc.)
> >> into
> >>> one feature matrix so that the analysis can use all the platforms
> >>> together.
> >>> the Broad firehose pipeline also has this as one of its outputs.
> >>>
> >>>
> >>>
> >>> as some of my comments allude to, it doesn't seem that Dataset
> >> Descriptions
> >>> deal with the use case of describing a dataset that is specifically
> >>> derived
> >>> from other datasets, which is what we are looking at ways we might
> >> describe
> >>> our data when we publish it.  i took a look at PROV and, i've got a
> >>> bit
> >>> more
> >>> mapping to do, but it seems like PROV provides the terms we need.
> >>>
> >>>
> >>>
> >>> but this has lead me to ask the question of what is the relation of
> >>> Dataset
> >>> Descriptions and PROV and how should they/should they be used
> >> together?  i
> >>> think the above use case is quite common for datasets being published
> so
> >>> might deserve a discussion in the Dataset Descriptions note
> >>>
> >>>
> >>>
> >>> cheers,
> >>>
> >>> michael
> >>>
> >>>
> >>>
> >>> Michael Miller
> >>>
> >>> Software Engineer
> >>>
> >>> Institute for Systems Biology
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> From: Joachim Baran [mailto:joachim.baran@gmail.com]
> >>> Sent: Thursday, July 31, 2014 3:43 PM
> >>> To: Michael Miller
> >>> Cc: w3c semweb hcls
> >>> Subject: Re: hcls dataset description comments
> >>>
> >>>
> >>>
> >>> Hi!
> >>>
> >>>
> >>>
> >>>  I will ponder about your edit suggestion of your first bullet point.
> >>> I
> >>> am
> >>> not sure at the moment if it would have wider implications.
> >>>
> >>>
> >>>
> >>>  You are right that the use cases were written by the groups
> >>> themselves. I
> >>> do not know how to improve the use cases without rewriting them,
> which
> >> might
> >>> not be agreeable to all parties involved. C'est la vie.
> >>>
> >>>
> >>>
> >>>  The role of Data Catalogs should then be discussed during out next
> >>> conf
> >>> call. Thanks for highlighting that this might be unclear to readers.
> >>>
> >>>
> >>>
> >>> Kim
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 30 July 2014 10:41, Michael Miller
> >>> <Michael.Miller@systemsbiology.org>
> >>> wrote:
> >>>
> >>> hi kim,
> >>>
> >>>
> >>>
> >>> 'For other edits, please fork the repository and create a pull request
> >>> with
> >>> your changes'
> >>>
> >>>
> >>>
> >>> of the four general comments, the first is really the only 'edit', i
> >>> didn't
> >>> put it in the minor edits because it had some implications that the
> >>> group
> >>> might not agree with.  if the change makes sense, it might be easier
> >>> for
> >>> you
> >>> to make the edit.
> >>>
> >>>
> >>>
> >>> the other three are general comments and i'm not sure what the
> solution
> >>> might be, they were mainly points, as a reader, that weren't clear or
> >>> were a
> >>> bit confusing.  these were all from the use case section so were
> >>> probably
> >>> written by the groups themselves?  if i have permission, i can
> >>> certainly
> >>> add
> >>> them as issues.
> >>>
> >>>
> >>>
> >>> cheers,
> >>>
> >>> michael
> >>>
> >>>
> >>>
> >>> Michael Miller
> >>>
> >>> Software Engineer
> >>>
> >>> Institute for Systems Biology
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> From: Joachim Baran [mailto:joachim.baran@gmail.com]
> >>> Sent: Tuesday, July 29, 2014 11:56 AM
> >>>
> >>>
> >>> To: Michael Miller
> >>> Cc: w3c semweb hcls
> >>> Subject: Re: hcls dataset description comments
> >>>
> >>>
> >>>
> >>> Hi!
> >>>
> >>>
> >>>
> >>>  Thanks for the suggestions. I have incorporated your minor edits.
> >>> Unbelievable how those slipped through after so many re-readings
> >>> still.
> >>>
> >>>
> >>>
> >>>  For other edits, please fork the repository and create a pull request
> >>> with
> >>> your changes.
> >>>
> >>>
> >>>
> >>> Best wishes,
> >>>
> >>>
> >>>
> >>> Kim
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 23 July 2014 08:53, Michael Miller
> >>> <Michael.Miller@systemsbiology.org>
> >>> wrote:
> >>>
> >>> hi kim,
> >>>
> >>>
> >>>
> >>> thanks for the pointer, i've updated my comments based on this newer
> >> draft
> >>> below.  many fewer and i especially like the complete example in 10.1!
> >>>
> >>>
> >>>
> >>> cheers,
> >>>
> >>> michael
> >>>
> >>>
> >>>
> >>> Michael Miller
> >>>
> >>> Software Engineer
> >>>
> >>> Institute for Systems Biology
> >>>
> >>>
> >>>
> >>> general comments:
> >>>
> >>> ·         s4.4 'Dataset Linking': might mention also that datasets are
> >>> derived from other datasets?
> >>> 'A dataset may incorporate, or link to, data in other datasets, e.g.
> >>> in
> >>> the
> >>> creation of a data mashup ' --> 'A dataset may incorporate, be derived
> >> from,
> >>> or link to, data in other datasets, e.g. in the analysis of original
> >>> datasets or in the creation of a data mashup '
> >>>
> >>> ·         s8: odd that some of the top sections (8.1-8.3,8.5-8.7) are
> >>> individual organizations but three (8.4, 8.8, 8.9) have subsections
> >>> for
> >>> different organizations.  maybe organize so all top level sections
> >>> define a
> >>> type of organization with subsections beneath or make all top-level?
> >>>
> >>> ·         s8: some of the use cases could be more focused on how this
> >>> note
> >>> will help them (8.5-8.7)
> >>>
> >>> ·         s8.9: how do Data Catalogs fit into this note?  wasn't clear
> >>> to me
> >>> how this note is relevant to them
> >>>
> >>> our use case questions:
> >>>
> >>> ·         how to reference 3rd party datasets that aren't described by
> >>> this
> >>> standard, i.e. TCGA data from the DCC, simply use 'pav:retrievedFrom'
> >>> with
> >>> the IRI being the URL into the repository?
> >>>
> >>> ·         we have a lot of intermediary files that we won't publish,
> >>> the
> >>> software specified in creating our published datasets from its sources
> >>> form
> >>> a (branching) workflow with the input being from the previous step(s)
> >>> in
> >> the
> >>> workflow.  how best to represent this?  this note doesn't seem to
> >>> cover
> >> how
> >>> the dataset is created so any recommendations?
> >>>
> >>> minor edits:
> >>>
> >>> ·         there are two s6.2.3 sections
> >>>
> >>> ·         s8.8.1: '... what period it is updated. To know when to...'
> >>> should
> >>> be '...what period it is updated to know when to...'?
> >>>
> >>>
> >>>
> >>> From: Joachim Baran [mailto:joachim.baran@gmail.com]
> >>> Sent: Tuesday, July 22, 2014 3:43 PM
> >>> To: Michael Miller
> >>> Cc: w3c semweb hcls
> >>> Subject: Re: hcls dataset description comments
> >>>
> >>>
> >>>
> >>> Hello,
> >>>
> >>>
> >>>
> >>>  I believe you were looking at an old document. There is currently
> >>> only
> >>> one
> >>> Figure in the note.
> >>>
> >>>
> >>>
> >>>  Please check the actual draft at:
> >>
>
http://htmlpreview.github.io/?https://github.com/joejimbo/HCLSDatasetDescriptions/blob/master/Overview.html
> >>>
> >>>
> >>>
> >>> Best wishes,
> >>>
> >>>
> >>>
> >>> Kim
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 22 July 2014 15:36, Michael Miller
> >>> <Michael.Miller@systemsbiology.org>
> >>> wrote:
> >>>
> >>> hi all,
> >>>
> >>>
> >>>
> >>> tremendous work, very clear and well-written.  my group at ISB, the
> >>> Shmulevich lab is looking to provide provenance for the analysis
> >>> datasets
> >> we
> >>> are producing for TCGA.  we're not sure if we'll be able to 'go all
> >>> the
> >>> way'
> >>> but we want to make sure we have at hand all the information that we
> >> could,
> >>> at least in theory, be compliant.  as long as i was reading the
> >>> document,
> >>> below are some notes.
> >>>
> >>>
> >>>
> >>> general comments:
> >>>
> >>> ·         s4.4 'Dataset Linking': might mention also that datasets are
> >>> derived from other datasets?
> >>> 'A dataset may incorporate, or link to, data in other datasets, e.g.
> >>> in
> >>> the
> >>> creation of a data mashup ' --> 'A dataset may incorporate, be derived
> >> from,
> >>> or link to, data in other datasets, e.g. in the analysis of original
> >>> datasets or in the creation of a data mashup '
> >>>
> >>> ·         the chembl example in s5 is not compliant to the property
> >>> table
> >>> below, it probably is only supposed to show the relationship of the
> >>> three
> >>> terms but that could be clarified
> >>>
> >>> ·         s6.2.12 could use the example filled in
> >>>
> >>> ·         6.3.2: not sure what an 'X level description' is
> >>>
> >>> ·         s8: odd that some of the top sections (8.1-8.3,8.5-8.7) are
> >>> individual organizations but three (8.4, 8.8, 8.9) have subsections
> >>> for
> >>> different organizations.  maybe organize so all top level sections
> >>> define a
> >>> type of organization with subsections beneath or make all top-level?
> >>>
> >>> ·         s8: many of the use cases could be more focused on how this
> >>> note
> >>> will help them
> >>>
> >>> ·         s8.9: how do Data Catalogs fit into this note?  wasn't clear
> >>> to me
> >>> how this note is relevant to them
> >>>
> >>> ·         would be nice to have a 'complete' example p[put together,
> >>> maybe
> >>> based on chembl?
> >>>
> >>>
> >>>
> >>> our use case questions:
> >>>
> >>> ·         how to reference 3rd party datasets that aren't described by
> >>> this
> >>> standard, i.e. TCGA data from the DCC, simply use 'pav:retrievedFrom'
> >>> with
> >>> the IRI being the URL into the repository?
> >>>
> >>> ·         we have a lot of intermediary files that we won't publish,
> >>> the
> >>> software specified in creating our published datasets from its sources
> >>> form
> >>> a (branching) workflow with the input being from the previous step(s)
> >>> in
> >> the
> >>> workflow.  how best to represent this?  this note doesn't seem to
> >>> cover
> >> how
> >>> the dataset is created so any recommendations?
> >>>
> >>>
> >>>
> >>> text issues:
> >>>
> >>> ·         Figure 1: 'Overview of dataset description level metadata
> >>> profiles
> >>> and their relationships': reference not resolved, image doesn't show
> >>>
> >>> ·         Figure 2: 'Improve diagram. Multiple appearance of
> >>> concepts/description levels unclear.': reference not resolved, image
> >> doesn't
> >>> show.  add actual label
> >>>
> >>>
> >>>
> >>> minor edits:
> >>>
> >>> ·         bottom of s.3: 'placeholde' should be 'placeholder'
> >>>
> >>> ·         use straight quotes rather than slant quotes in s6.2.2
> >>> example
> >>> (and elsewhere)?
> >>>
> >>> ·         the text runs out of the box in s6.2.3 'Description'
> >>>
> >>> ·         s6.2.3: 'Dates of Creation and Issuance': 'state the date
> >>> the
> >>> dataset was generated using dct:created and/or the date the dataset
> was
> >> made
> >>> public using dct:created' should be 'state the date the dataset was
> >>> generated using dct:created and/or the date the dataset was made
> public
> >>> using dct:issued'?
> >>>
> >>> ·         there are two s6.2.3 sections
> >>>
> >>> ·         s6.2.4: 'Creation: ... The date of authorship' should be
> >>> '...The
> >>> date of creation' and 'Curation:... The date of authorship' should be
> >>> '...The date of curation'?
> >>>
> >>> ·         s8.5: the author list has end parenthesis without beginning
> >>> parenthesis
> >>>
> >>> ·         s8.8.1: '... what period it is updated. To know when to...'
> >>> should
> >>> be '...what period it is updated to know when to...'
> >>>
> >>>
> >>>
> >>> cheers,
> >>>
> >>> michael
> >>>
> >>>
> >>>
> >>> Michael Miller
> >>>
> >>> Software Engineer
> >>>
> >>> Institute for Systems Biology
> >>
> >>
> >>
> >> --
> >> Stian Soiland-Reyes, myGrid team
> >> School of Computer Science
> >> The University of Manchester
> >> http://soiland-reyes.com/stian/work/ http://orcid.org/0000-0001-9842-
> 9718