Re: New open source catalog and list of APIs on Data.gov

Having constructed JSONP files by hand for: http://www.rustprivacy.org/2013/egov/penserv/
"cannot"  = "ginormously tedious" = csv is a real, real good idea (approx.)
--Gannon

________________________________
 From: Josh Tauberer <tauberer@govtrack.us>
To: Brand Niemann <bniemann@cox.net> 
Cc: 'Irina Bolychevsky' <irina.bolychevsky@okfn.org>; "'Holm, Jeanne M (1760)'" <jeanne.m.holm@jpl.nasa.gov>; 'eGov W3C' <public-egov-ig@w3.org>; 'CKAN discuss' <ckan-discuss@lists.okfn.org>; 'CKAN Development Discussions' <ckan-dev@lists.okfn.org> 
Sent: Thursday, May 23, 2013 4:33 PM
Subject: Re: New open source catalog and list of APIs on Data.gov
 


On 05/23/2013 01:55 PM, Brand Niemann wrote:

So if I am the public/data scientist, I cannot really use:
I think it would be more useful for this group to stick to
      constructive questions like "what is the best way to X?" rather
      than statements like "I cannot."

So, of course you can use the files. Just find a programmer friend
      who can spend maybe five minutes converting the file for you. Or
      Google for "json to csv" to find some ways other people have
      solved this problem.

(Of course it would be great if CKAN added a CSV export in version
      2.2. Or if someone submitted a patch to do it. It's an open source
      project, after all.)



- Josh Tauberer (@JoshData) http://razor.occams.info
On 05/23/2013 01:55 PM, Brand Niemann wrote:

 
>Josh, Many thanks for the explanation-application. Yes, the CKAN API is running very slow and yes, I got nearly immediate response from your mirror sites. Now what do I do with 25 MB and 8 KB Jason files? I want something I can readily work with in Spotfire like CSV, Excel, etc.
> 
>So if I am the public/data scientist, I cannot really use:
> 
>http://catalog.data.gov where I would have to look at and use 3683 pages
> 
>and 
> 
>https://catalog.data.gov/api/3/action/package_list where it takes so long and returns a format that I cannot import into Spotfire
> 
>If I had the entire catalog I could expand my Spotfire application:
> 
>https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?OpenGovernmentData-Spotfire
> 
>to search and readily download from the catalog, a specific data set of interest and import it into Spotfire and visualize it like I have done for the Federal Data Center Consolidation Data Center Closings 2010-2013.xlsx
> 
>Irina, This answers your suggestion and question:
> 
>You can download the files directly from search result listing by clicking the format icons. Or from dataset pages. If you're looking to download datasets on bulk, you can do this through the CKAN API, in blocks of 100 for example. API calls detailed here: http://docs.ckan.org/en/ckan-2.0/api.html
> 
>What are you looking to do?
> 
>Brand
> 
>From:Josh Tauberer [mailto:tauberer@govtrack.us] 
>Sent: Thursday, May 23, 2013 11:31 AM
>To: Brand Niemann
>Cc: 'Holm, Jeanne M (1760)'; 'eGov W3C'
>Subject: Re: New open source catalog and list of APIs on Data.gov
> 
>Brand,
>
>Since the new site is based on CKAN, you can read the CKAN
            API documentation yourself to figure out how to download the
            data catalog:
>
>    http://docs.ckan.org/en/ckan-2.0/api.html
>
>The API is running really slow right now (probably all of us
            checking out the new site), so I'd suggest *not* hitting the
            API link below right now, but I've mirrored it so you can
            see the output quickly:
>
>https://catalog.data.gov/api/3/action/package_list
>  => Produces a JSON list of entries in the catalog
>  => Here's a static mirror: http://razor.occams.info/files/catalog_data_gov_api_action_package_list.json
>
>The package_search API function returns complete metadata
            rather than just a list of package names. It requires an
            HTTP POST, so here's an example of that:
>
>    curl -d '{"rows":1, "start": 0}' https://catalog.data.gov/api/3/action/package_search
>    => static mirror: http://razor.occams.info/files/catalog_data_gov_api_action_package_search.json
>
>
>
>- Josh Tauberer (@JoshData)
> 
>http://razor.occams.info
>On 05/23/2013 10:34 AM, Brand Niemann wrote:
>Thanks, Jeanne. Please send me the link to download the entire and new data catalog.
>> 
>>I am getting ready for: On Thursday, May 23rd at 2:00PM ET, Federal Chief Information Officer Steven VanRoekel will host a conference call to discuss the 1-yr Anniversary of the Digital Government Strategy and ongoing Administration efforts to drive innovation through open data and other initiatives.
>> 
>>Brand
>> 
>>From:Holm, Jeanne M (1760) [mailto:jeanne.m.holm@jpl.nasa.gov] 
>>Sent: Thursday, May 23, 2013 10:10 AM
>>To: Brand Niemann; 'eGov W3C'
>>Subject: Re: New open source catalog and list of APIs on Data.gov
>> 
>>Brand--
>> 
>>As John pointed out, the entire and new catalog (as of this morning) is available at https://catalog.data.gov/dataset 
>> 
>>The catalog at https://explore.data.gov/Other/Data-gov-Catalog/pyv4-fkgv is our previous catalog of "raw" data that is not part of the larger catalog above.
>> 
>>The number differs from our earlier number of datasets:  The total number of datasets reflects datasets plus data series. A data series may contain a large number of additional products or files of the same type. The previous Data.gov catalog counted the individual datasets and not just the series and therefore had a higher total number of datasets noted. The change in number does not reflect a change in the size of the catalog, but rather a different structure of data products in the new catalog.
>> 
>>Hope this helps to address any confusion.
>> 
>>--Jeanne
>>
>>**********************************************************
>>Jeanne Holm
>>Evangelist, Data.gov
>>U.S. General Services Administration
>>Cell: (818) 434-5037
>>Twitter/Facebook/LinkedIn: JeanneHolm
>>**********************************************************
>> 
>>
>>________________________________
>>
>>From:Brand Niemann <bniemann@cox.net>
>>To: "'Holm, Jeanne M (1760)'" <jeanne.m.holm@jpl.nasa.gov>; 'eGov W3C' <public-egov-ig@w3.org> 
>>Sent: Thursday, May 23, 2013 8:10 AM
>>Subject: RE: New open source catalog and list of APIs on Data.gov
>> 
>>Jeanne, Thank you. I heard Doug Nebert announce this yesterday at the UCGIS 2013 Symposium:
>>http://ucgis2.org/event-item/preliminary-program
>> 
>>When I look for the 73,651 data sets, I find only 7,808 at: https://explore.data.gov/Other/Data-gov-Catalog/pyv4-fkgv
>> 
>>So where are the other 65,843?
>> 
>>My audit for reproducible results is at: http://semanticommunity.info/An_Open_Data_Policy
>> 
>>Thanks, Brand
>> 
>>Dr. Brand Niemann
>>Director and Senior Data Scientist
>>Semantic Community
>>http://semanticommunity.info
>>http://gov.aol.com/bloggers/brand-niemann/
>>703-268-9314
>> 
>>From:Holm, Jeanne M (1760) [mailto:jeanne.m.holm@jpl.nasa.gov] 
>>Sent: Thursday, May 23, 2013 8:42 AM
>>To: eGov W3C
>>Subject: New open source catalog and list of APIs on Data.gov
>> 
>>Hi all--
>> 
>>I invite you to visit Data.gov to see the new catalog for browsing U.S. open data: http://catalog.data.gov We have combined raw and geospatial data from many sources across the U.S. and presented it through an open source tool, CKAN. 
>> 
>>In connection with the U.S. Digital Strategy we have also created a new list of government APIs: http://www.data.gov/developers/page/developer-resources 
>> 
>>Find out more at: http://www.data.gov/blog/datagov-launches-new-catalog-and-apis  
>> 
>>--Jeanne Holm
>>
>>**********************************************************
>>Jeanne Holm
>>Evangelist, Data.gov
>>U.S. General Services Administration
>>Cell: (818) 434-5037
>>Twitter/Facebook/LinkedIn: JeanneHolm
>>**********************************************************
>> 
>