Re: [sunlightlabs] Open Government Data Standards

FYI. You can view the thread online at https://groups.google.com/forum/?fromgroups=#!topic/sunlightlabs/kI7V8jm8ISk

On 2013-03-20, at 5:02 PM, Philip Ashlock wrote:

> I responded to a few things here, but maybe it'd be better to try to move the discussion over on the public-opengov list (http://lists.w3.org/Archives/Public/public-opengov/)
> 
> On Wed, Mar 20, 2013 at 3:45 PM, James McKinney <james@opennorth.ca> wrote:
> Hi Tom, I've copied your questions and replied inline.
> 
>> - I know geography's on the Popolo to do list, and of course I know you've done a ton of work in this area with the Open States team already. It'd be great to hear your thinking, though -- NOI quickly convinced us that this was going to be a tricky but absolutely essential area for muni work.
> 
> My initial research is at https://github.com/opennorth/popolo-standard/issues/21
> 
> Not all questions relating to geography are tricky. Can you present some of the specific issues you've identified relating to geography?
> 
> I don't mean to speak for Tom here, but my guess is that he was referring to the tricky work of attempting to use geographic tools and data models for data that isn't entirely defined geospatially. A good example of this are voting precints which are often defined by address ranges including those that might bisect an apartment complex where you would really start to need a 3 dimensional GIS system to accurately model that as a geography :) In fact many things that we think of as boundaries are merely attempts to convert address ranges into polygons, but there can be all sorts of accuracy and precisions issues that come up through that conversion.
>  
> 
> To pick one problem, some people try to classify areas by "administrative level". This breaks down, because some countries have deep hierarchies and others don't, and it quickly becomes impossible to compare levels between countries or to attach any sort of meaning to a specific level, besides the trivial "this level is deeper than level 5 and shallower than level 7". So, I don't recommend numbered administrative levels.
> 
> Anyway, I look forward to more specific questions, so that I can provide more interesting answers!
> 
> 
> I don't think attempting to define comprehensive administrative "hierarchies" should be a priority for anyone to try to coordinate, assuming they even could be modeled accurately, but it's certainly something that people will intuitively bring up. Many parts of government are normally thought of with some sense of a hierarchy, so I don't think we should be afraid to acknowledge that and include whatever data might exist that connotes that, but I think it would be foolish to think that we could model that in any kind of consistent or universal way. I do still think it would be valuable to have a reserved field to describe the colloquial levels of government in this perceived hierarchy even if the values in that field were not part of a strictly controlled vocabulary. I generally think it's more useful to view the idea of administrative "hierarchies" as being an attribute that describes a relationship relative to direct child/parent entities in a very independent way - rather than in any kind of predefined universal hierarchy that could be used internationally. It's valuable to know that a city council district is a "child" of a city even if the notion of a "city council district" isn't something that exists everywhere and even if you don't define any parent entity associated with the city or any child entity associated with the city council district. 
> 
> This is probably just another way of saying what others have already said, but just wanted to convey my way of thinking about it. 
>  
> 
>> - the taxonomy implicit in 2.2.5 is the other big piece -- seems like a hugely important and complicated one, but pretty essential for a number of questions. My sense is that we're just going to have to fight with the data until patterns start to jump out, but is there another approach you're considering? DemocracyMap has a good starting list here but I suspect we're going to see a bunch more weirdness before all is said & done.
> 
> 
> When it comes to controlled vocabularies (a useful keyword), I try to go with what already exists. Some countries have very well defined, government-maintained lists for classifying areas, organizations, people, etc. It often takes a fair bit of time to track them down. Generally, these classifications schemes will be geography-specific. It would be very hard to find/define a universal, international standard that wouldn't leave out a lot of detail that specific geographies care about.
> 
> So, the Popolo spec defers on that question, but I would be interested in providing a list of existing classification schemes to choose from. 
> 
> 
> 
> Is there a good middle path? Can we explore a strategy whereby we leverage the well defined, more universal controlled vocabularies where possible and provide room for the location specific data within an attached schemaless data object (an "extra fields" list or the way OpenStates uses the occasionally available +fields), but try to observe how that kind of sandbox is used to see if there are repeated patterns that can eventually be brought into the more controlled vocabulary?
> 
> 
>  
> James
> 
> 
> On 2013-03-20, at 3:15 PM, Tom Lee wrote:
> 
>> Congratulations on getting the spec to this point, James! As you know (but others might not), Sunlight is beginning municipal data collection thanks to our partnership with Google and NOI's BIP/VIP projects. This has been in the works for a while (Open States has supported DC for a long time, after all), but we're pretty excited to push things up to new scales. (We're looking forward to soliciting feedback about these plans from the larger community, but we're still squaring away how we can collect things in parallel without creating tons of collisions -- more details on that soon, I hope.)
>> 
>> In general, though, I've been thinking about this problem through the lens of the Open States playbook, which emphasizes schemalessness and the inevitability of refactoring the model once data starts to come in. But the more people thinking about this stuff the better (and James T might have a different and better-informed sense). And if Popolo brings PopIt support with it, then obviously supporting it as an export format becomes very attractive.
>> 
>> Two questions for you:
>> 
>> - I know geography's on the Popolo to do list, and of course I know you've done a ton of work in this area with the Open States team already. It'd be great to hear your thinking, though -- NOI quickly convinced us that this was going to be a tricky but absolutely essential area for muni work.
>> 
>> - the taxonomy implicit in 2.2.5 is the other big piece -- seems like a hugely important and complicated one, but pretty essential for a number of questions. My sense is that we're just going to have to fight with the data until patterns start to jump out, but is there another approach you're considering? DemocracyMap has a good starting list here but I suspect we're going to see a bunch more weirdness before all is said & done.
>> 
>> 
>> On Tue, Mar 19, 2013 at 11:25 PM, James McKinney <james@opennorth.ca> wrote:
>> I'd like to announce to this group a new community project aimed at people creating civic technology and (re-)publishing government data to adopt standards for their data and APIs: the Popolo project.
>> 
>> http://popoloproject.com/
>> 
>> A major barrier to increased re-use of the growing number of open-source civic tools is the lack of agreement on how to name things. To give a very simple example: if one project's elected officials API calls a person’s name "name" and another calls it "full_name", and you're writing a Q&A platform to ask questions to these elected officials, you'll need to write an adapter for each API. Committing to a standard way of naming things would maximize interoperability, reduce wheel reinvention and make re-use that much easier.
>> 
>> The project's process is to (1) come up with use cases and requirements (for example, find an elected official by postal address), (2) identify existing standards addressing those use cases and requirements and (3) write specifications for how to combine and re-use those existing standards in a standard way, filling the gaps between those standards when necessary. The current spec addresses how to store/share information about people, organizations and memberships, and will soon expand to areas (e.g. districts) and events (e.g. elections).
>> 
>> This is a consensus-based, community-driven project, so we are eager to receive your feedback and contributions on the draft spec and for you to help define and start work on new specs with the support of the group. A W3C Open Government Community Group (CG) has been created to host the community around the specs:
>> 
>> http://www.w3.org/community/opengov/
>> 
>> Discussions happen through the CG mailing list at:
>> 
>> http://lists.w3.org/Archives/Public/public-opengov/
>> 
>> To be clear, the Popolo project, for which I am responsible, covers only a subset of the specs relevant to open government data. Its general scope is data relating to the legislative branch. Health inspections data, for example, covered by Yelp's LIVES spec, would be out of scope of Popolo. Within its scope, its focus is on data that often appears together and that multiple sources publish; for example, Open States publishes data on people, committees (organizations), bills (documents), votes and events, as do many other projects.
>> 
>> In terms of adoption and community, mySociety is working towards aligning PopIt (their people-organizations-positions web service) with Popolo. The people at the Sunlight Foundation have also been providing great feedback.
>> 
>> Through the CG mailing list linked above, I encourage those of you who consume data to submit new use cases and requirements, and those of you who publish data to provide feedback on the draft spec. Everyone can help decide what new specs the group should focus its efforts on, and to work on those following the rough three steps described above.
>> 
>> To be clear, the "Popolo" name is only tied to the spec which I am the editor of, and the Popolo spec is just one of the specs that the CG can come up with. The CG is meant to be a shared workspace for open government data spec editors.
>> 
>> Last few notes:
>> 
>> In order to support the research, development, maintenance and improvement of the Popolo spec and the outreach and facilitation of the community group, I've submitted the following to the Knight News Challenge. The News Challenge is in its "feedback" phase for the next ten days, so I look forward to your comments!
>> 
>> https://www.newschallenge.org/open/open-government/submission/legislative-open-government-data-standards/
>> 
>> The Popolo spec is managed on GitHub where you are welcome to report specific issues:
>> 
>> https://github.com/opennorth/popolo-standard/tree/gh-pages
>> 
>> If you will be attending Transparency Camp, please comment and vote up the proposed session about data standards at
>> 
>> http://transparencycamp.org/ideas/10/
>> 
>> If you have any questions, please let me know either on this list or the CG mailing list at http://lists.w3.org/Archives/Public/public-opengov/
>> 
>> Best,
>> 
>> --
>> James McKinney
>> http://opennorth.ca/
>> 
>> --
>> You received this message because you are subscribed to the Google Groups "sunlightlabs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to sunlightlabs+unsubscribe@googlegroups.com.
>> To post to this group, send email to sunlightlabs@googlegroups.com.
>> Visit this group at http://groups.google.com/group/sunlightlabs?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>> 
>> 
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups "sunlightlabs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to sunlightlabs+unsubscribe@googlegroups.com.
>> To post to this group, send email to sunlightlabs@googlegroups.com.
>> Visit this group at http://groups.google.com/group/sunlightlabs?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>  
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups "sunlightlabs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sunlightlabs+unsubscribe@googlegroups.com.
> To post to this group, send email to sunlightlabs@googlegroups.com.
> Visit this group at http://groups.google.com/group/sunlightlabs?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups "sunlightlabs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sunlightlabs+unsubscribe@googlegroups.com.
> To post to this group, send email to sunlightlabs@googlegroups.com.
> Visit this group at http://groups.google.com/group/sunlightlabs?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>