A Data Provider Tool for Data.gov

Who on the Open Government Data Science Team is responsible for sucking up to the Boss ?  Today The White House announced five executive actions and seven 
recommendations for Congress to protect innovators from lawsuits, or so I'm told.

I made an index of Agencies and Subdivisions of record (177) for the current catalog download, and a cross-reference of the Keyword Sets used, by Agency.  It is in the Catalog Zip File[*] or here: http://tinyurl.com/agency-keywords

The intent is to give Agencies who are preparing documents for Data.gov some indication of the Keywords used before.  This is a SEO type function, and unrelated to "Linked Data" (Custom Metadata provided by the Agency), but it is something an Agency can do to promote message coherence, reproducibility and sucking up (see above).

Brand, you should be able to copy and paste the HTML (Index) Table into a spreadsheet.  I did clean up duplicate names, eliminated TLA's (they are a Linked Data encoding nightmare) and unicode escape entities.  If you have any problems, let me know.

--Gannon

[*] http://www.rustprivacy.org/2013/egov/catalog/DataDotGovCatalog.zip




________________________________
 From: Brand Niemann <bniemann@cox.net>
To: 'Gannon Dick' <gannon_dick@yahoo.com>; "'Holm, Jeanne M (1760)'" <jeanne.m.holm@jpl.nasa.gov>; 'eGov W3C' <public-egov-ig@w3.org> 
Sent: Sunday, June 2, 2013 5:27 AM
Subject: RE: New open source catalog and list of APIs on Data.gov
 


Thanks again Gannon!
 
We have an Open Government Data Data Science Team as follows:
Steven VanRoekel - Federal CIO - Directs the Digital Government Strategy
Jeanne Holm - Data.gov Evangelist - Evangelizes the Availability of the Data
Gannon Dick - Data Preparation - Prepares the Data for Analysis
Brand Niemann - Data Scientist - Provides the Data (Catalog and Results) in a Data Platform
 
http://semanticommunity.info/An_Open_Data_Policy#Story
 
Dr. Brand Niemann
Director and Senior Data Scientist
Semantic Community
http://semanticommunity.info 
http://gov.aol.com/bloggers/brand-niemann/ 
703-268-9314
 
From:Gannon Dick [mailto:gannon_dick@yahoo.com] 
Sent: Saturday, June 01, 2013 3:54 PM
To: Brand Niemann; 'Holm, Jeanne M (1760)'; 'eGov W3C'
Subject: Re: New open source catalog and list of APIs on Data.gov
 
You are welcome, Dr. Niemann.

In the meantime, I realized I could do something interesting with the catalog having chopped it up into fields.

There are keywords (dct:subject in Dublin Core parlance).  They are in a comma delimited list (like a CURI - Compact URI, although they don't have a namespace).  But there is no reason not to make a list of individual terms by Agency.  The catalog also contains "Custom Metadata" from the Agencies.  I collated that too.  In addition, I was doing some work with the FACA Database and made a list of identifiers for 2011 (it has the Agency URLs, Acronym, etc.).

Spreadsheets are in http://www.rustprivacy.org/2013/egov/catalog/DataDotGovCatalog.zip

If you just want to take a quick look ... http://www.rustprivacy.org/2013/egov/catalog/AgencyMetadataPopulated.csv

--Gannon
 
 

________________________________

From:Brand Niemann <bniemann@cox.net>
To: "'Holm, Jeanne M (1760)'" <jeanne.m.holm@jpl.nasa.gov>; 'eGov W3C' <public-egov-ig@w3.org> 
Cc: Gannon Dick <gannon_dick@yahoo.com> 
Sent: Friday, May 31, 2013 8:21 PM
Subject: RE: New open source catalog and list of APIs on Data.gov
 
Jeanne, Thank you. The Data.gov Catalog and a specific example (Federal Data Consolidation Initiative) and the New Data.gov Catalog (thanks to Gannon) and a specific example (USGS Sediment Texture Database) using the same Data Platform (Spotfire) for the catalogs and the examples are found at:
 
http://semanticommunity.info/An_Open_Data_Policy
 
http://semanticommunity.info/GIS_Data_Science
 
Dr. Brand Niemann
Director and Senior Data Scientist
Semantic Community
http://semanticommunity.info 
http://gov.aol.com/bloggers/brand-niemann/ 
703-268-9314
 
From:Holm, Jeanne M (1760) [mailto:jeanne.m.holm@jpl.nasa.gov] 
Sent: Thursday, May 30, 2013 2:49 PM
To: Brand Niemann; 'eGov W3C'
Subject: Re: New open source catalog and list of APIs on Data.gov
 
Hi all--
 
Here's the way to download the entire catalog.  The Data.gov catalog (exclusive of geospatial data) can still be accessed at https://explore.data.gov/Other/Data-gov-Catalog/pyv4-fkgv  You can use the export button at the top right to be able to download it in a variety of formats.
 
The API is another way to access the entire catalog.
 
We are exploring other ways to make the newly integrated catalog available.
 
--Jeanne

**********************************************************
Jeanne Holm
Evangelist, Data.gov
U.S. General Services Administration
Cell: (818) 434-5037
Twitter/Facebook/LinkedIn: JeanneHolm
**********************************************************
 
From: Brand Niemann <bniemann@cox.net>
Date: Thursday, May 23, 2013 7:34 AM
To: Jeanne Holm <Jeanne.M.Holm@jpl.nasa.gov>, eGov W3C <public-egov-ig@w3.org>
Subject: RE: New open source catalog and list of APIs on Data.gov
 
Thanks, Jeanne. Please send me the link to download the entire and new data catalog.
 
I am getting ready for: On Thursday, May 23rd at 2:00PM ET, Federal Chief Information Officer Steven VanRoekel will host a conference call to discuss the 1-yr Anniversary of the Digital Government Strategy and ongoing Administration efforts to drive innovation through open data and other initiatives.
 
Brand
 
From:Holm, Jeanne M (1760) [mailto:jeanne.m.holm@jpl.nasa.gov] 
Sent: Thursday, May 23, 2013 10:10 AM
To: Brand Niemann; 'eGov W3C'
Subject: Re: New open source catalog and list of APIs on Data.gov
 
Brand--
 
As John pointed out, the entire and new catalog (as of this morning) is available at https://catalog.data.gov/dataset 
 
The catalog at https://explore.data.gov/Other/Data-gov-Catalog/pyv4-fkgv is our previous catalog of "raw" data that is not part of the larger catalog above.
 
The number differs from our earlier number of datasets:  The total number of datasets reflects datasets plus data series. A data series may contain a large number of additional products or files of the same type. The previous Data.gov catalog counted the individual datasets and not just the series and therefore had a higher total number of datasets noted. The change in number does not reflect a change in the size of the catalog, but rather a different structure of data products in the new catalog.
 
Hope this helps to address any confusion.
 
--Jeanne

**********************************************************
Jeanne Holm
Evangelist, Data.gov
U.S. General Services Administration
Cell: (818) 434-5037
Twitter/Facebook/LinkedIn: JeanneHolm
**********************************************************
 

________________________________

From:Brand Niemann <bniemann@cox.net>
To: "'Holm, Jeanne M (1760)'" <jeanne.m.holm@jpl.nasa.gov>; 'eGov W3C' <public-egov-ig@w3.org> 
Sent: Thursday, May 23, 2013 8:10 AM
Subject: RE: New open source catalog and list of APIs on Data.gov
 
Jeanne, Thank you. I heard Doug Nebert announce this yesterday at the UCGIS 2013 Symposium:
http://ucgis2.org/event-item/preliminary-program
 
When I look for the 73,651 data sets, I find only 7,808 at: https://explore.data.gov/Other/Data-gov-Catalog/pyv4-fkgv
 
So where are the other 65,843?
 
My audit for reproducible results is at: http://semanticommunity.info/An_Open_Data_Policy
 
Thanks, Brand
 
Dr. Brand Niemann
Director and Senior Data Scientist
Semantic Community
http://semanticommunity.info
http://gov.aol.com/bloggers/brand-niemann/
703-268-9314
 
From:Holm, Jeanne M (1760) [mailto:jeanne.m.holm@jpl.nasa.gov] 
Sent: Thursday, May 23, 2013 8:42 AM
To: eGov W3C
Subject: New open source catalog and list of APIs on Data.gov
 
Hi all--
 
I invite you to visit Data.gov to see the new catalog for browsing U.S. open data: http://catalog.data.gov We have combined raw and geospatial data from many sources across the U.S. and presented it through an open source tool, CKAN. 
 
In connection with the U.S. Digital Strategy we have also created a new list of government APIs: http://www.data.gov/developers/page/developer-resources 
 
Find out more at: http://www.data.gov/blog/datagov-launches-new-catalog-and-apis  
 
--Jeanne Holm

**********************************************************
Jeanne Holm
Evangelist, Data.gov
U.S. General Services Administration
Cell: (818) 434-5037
Twitter/Facebook/LinkedIn: JeanneHolm
**********************************************************