Re: Absence of key scientific spatial data formats within common formats to implementation of Best Practices [SEC=UNCLASSIFIED]

Yes fully accept the "relative" part,  but we are talking about "the web"
as the user community and that is even bigger than Google ;-)

Ed


On Wed, 30 Mar 2016 at 09:43 Bill Roberts <bill@swirrl.com> wrote:

> Hi Lewis
>
> Thanks for the very useful input on scientific data formats.  I don't
> remember exactly the background to the list of spatial formats that was the
> starting point for this thread, but although those formats like GML,
> GeoJSON etc can encode 'data', I (and perhaps others?) tend to think of
> those as formats for 'geometry' whereas I think of NetCDF as a format for
> 'data'.
>
> Anyway, I don't want to get into that, I just want to note that I'm one of
> the editors of the Coverages sub-group and the points you have raised in
> recent emails are very relevant for that.  The main issues we are
> discussing in that group relate to 'web-friendly' formats for coverage data
> (whatever web-friendly turns out to mean in that context!) and approaches
> to identifying and retrieving extracts (aka subsets) of Coverage data.
>
> For both of those, it would be great to get your input and to make sure we
> take due consideration of use cases that are important to JPL, NASA etc.
>
> Cheers
>
> Bill
>
>
>
> On 30 March 2016 at 07:38, lewis john mcgibbney <lewismc@apache.org>
> wrote:
>
>> Hi Bruce,
>> Replies inline
>>
>> On Mon, Mar 28, 2016 at 7:21 PM, Bruce Bannerman <B.Bannerman@bom.gov.au>
>> wrote:
>>
>>>
>>>
>>> Hi Lewis,
>>>
>>> More inline below.
>>>
>>> Bruce
>>>
>>>
>>> Regarding describing our datasets:
>>>
>>>    - We don’t do this as well as we could. We will begin addressing
>>>    this in the near future. Much of our current work is either internally
>>>    focussed, or at a much too granular level.
>>>
>>>
>> Understood. This also seems to be quite a common observation from the
>> datasets (and the platforms which make this data available) I work with so
>> I acknowledge the point.
>>
>>>
>>>    - We intend describing our data sets using ISO 19115 with support
>>>    for several profiles, including WMO and ANZLIC.
>>>
>>> I believe the OCO2 products (and many others) I've worked with also came
>> with ISO 19115 metadata within the data product. These products were HDF5.
>> I am familiar with the ISO standard(s) as well.
>> http://oco.jpl.nasa.gov/science/ProductInfo/#
>>
>>
>>>
>>>    - I can’t see us moving away from this paradigm, but there is
>>>    certainly potential for LinkedData approaches as alternate methods of
>>>    discovering our data.
>>>
>>> Agreed. This is where my work in this group is (again) justified.
>>
>>
>>> But this is also only part of the issue:
>>>
>>>    - We also need a mechanism to better understand the context of our
>>>    observations (e.g. What sensor; what model; when was it last calibrated;
>>>    maintained; what sensor maintenance process and responsible party; what
>>>    observation process etc). We will be using the new WMO WIGOS Observations
>>>    Metadata standard to support this concept.
>>>
>>> Interesting. Have you looked at encoding this into any linked data
>> approach as of yet?
>>
>>
>>>
>>>    - As discussed before on this list (and in the SDWWG Climate data
>>>    related use case), there is also the issue of data provenance.
>>>
>>> Yes there sure is. We make heavy use of the PROV-ES specification here
>> @JPL for such requirements.
>>
>>>
>>>    - Data Quality and IP issues will also become a big issue,
>>>    particularly with the increasing use of mixed Bureau and 3rd party
>>>    observations and the subsequent derived products that we create from these
>>>    observations.
>>>
>>> Derived and value added products are certainly in high demand for
>> new(er) data products which are made available, however without an
>> expansion on this topic I see this more as a process issue rather than one
>> relating to the spatial data format itself. This is absolutely OK though.
>> If you feel like expanding then I am all ears. I see that such topics
>> feature heavily within the CDMS spec you posted below. These are relevant
>> topics indeed however I'll state that I am not sure they feature on the
>> current agenda for this WG.
>>
>>
>>>
>>>
>>>
>>>> Further, I expect that we'll need to go further and work with our peers
>>>> to agree on semantic definitions of the content that we portray for each
>>>> relevant domain and its inter-relationships with other domains.
>>>>
>>>
>>> This sounds like the next step... the issues we're discussing above seem
>>> like the precursor. Am I correct?
>>>
>>>
>>> Not necessarily, consider the work that has been undertaken on GeoSciML,
>>> WaterML etc.
>>>
>>> A lot of this is based on communities of a common interest getting
>>> together and agreeing on and using common terms and concepts.
>>>
>>> It takes many, many years of community building to reach the required
>>> consensus.
>>>
>>
>> OK, I was just trying to bring it back to what we can achieve within the
>> maneuverability and scope of this WG.
>>
>>
>>>
>>>
>>> Do you have any examples from the field of Meteorology? i would be
>>> interested to see if I could pick out any examples more familiar to other
>>> aspects of Earth Science, Pysical Oceanography or something else a bit
>>> closer to 'home' for my current working agenda.
>>>
>>>
>>>
>>> The closest that I can point to at the moment is the work that we have
>>> been doing in WMO on WMO #1131, Climate Data Management System
>>> Specifications h
>>> ttp://library.wmo.int/opac/index.php?lvl=notice_display&id=16300
>>> <http://library.wmo.int/opac/index.php?lvl=notice_display&id=16300>
>>>
>>
>> Wow this is a meaty, very substantial document. It will take me a while
>> to read as it's the first time I've seen it. I undertook a preliminary
>> search for 'data access' and 'access' and it returned a few results so I
>> will scope them out and see what interesting content I can muse over.
>>
>>
>>>
>>> There is also related work, e.g.:
>>>
>>>    - Foundation data governance and data modelling work within WMO that
>>>    Jeremy Tandy is leading
>>>    - Foundation work that has been undertaken by Australia’s CSIRO over
>>>    many years: https://www.seegrid.csiro.au/wiki/Siss/WebHome
>>>    - And to be honest, much of the underpinning OGC standards efforts
>>>    that we build on top of.
>>>
>>>
>>> This is really laying the groundwork, and it will take many years to get
>>> there with truly federated data and data services.
>>>
>>
>> So what are your thoughts then about how this all fits in with one or
>> more of the aims of this WG? When worded like it has been above, this
>> scientifici data angle (which you, I and a few others are coming from)
>> seems to be somewhat different from the other working group members. It is
>> certainly a different conversation we are having here from what I have seen
>> or heard going on elsewhere in this WG. I've also checked the WG mailing
>> list archives are there is very little conversation at all about scientific
>> data formats within the overall context of this WG.
>>
>> Thanks. I am glad to see that this thread is now picking up some traction.
>> Lewis
>>
>
> --

*Ed Parsons *FRGS
Geospatial Technologist, Google

Google Voice +44 (0)20 7881 4501
www.edparsons.com @edparsons