Re: [odcgeneralstewardslist] IODC+ DATA ROADMAPS: Gov of Mex on Bundled Commitment on IODC & National Consultation/Roadmap on the Data Revolution for Sustainable Development

Jose,

I don't think we need to organize more sessions at conferences to figure
out how to measure OD utilization.  Might be easier to just talk to
Socrata, Junar, NuCivic, and CKAN folks to organize common utilization
metrics.

I am adding my W3C Data on the Web Best Practices Working Group colleagues
to this discussion since standardizing that metadata is within our mandate
and we are working on Data Quality and Usability Vocabularies.

Also adding Kevin Merrit (Socrata), Diego May (Junar), and Andrew Hoppin
(NuCivic).  I guess there are people already on copy who can represent
CKAN.

Kevin, Diego, Andrew - We are having a conversation about how to measure
aggregate Open Data utilization and we wonder if it would be possible to
agree on common metadata standards that would allow API calls to your OD
catalogs.  We would like to be able to add OD utilization and quality
statistics to common OD Supply Indexes.

Sorry to dump you all into this long thread.  But it seems to me that right
now, while our industry is relatively small, we have the opportunity to
agree on common standards that could really benefit many interests.

Could we ask you for your views on this topic?

Best Regards,

Steve

Motto: "Do First, Think, Do it Again"


|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"Jose M. Alonso" <josema@webfoundation.org>                                                                                                       |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Tim Davies <tim@practicalparticipation.co.uk>                                                                                                     |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Cc:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Steven Adler/Somers/IBM@IBMUS, Daniel Dietrich <daniel.dietrich@okfn.org>, Barbara Ubaldi <Barbara.UBALDI@oecd.org>, Marcio Vasconcelos           |
  |<Marcio.Vasconcelos@avina.net>, "ODCstewardslist@opendatacharter.net" <odcstewardslist@opendatacharter.net>, sumandro <sumandro@cis-india.org>,   |
  |"Zeitz, Paul S" <ZeitzPS@state.gov>, Carlos Iglesias <carlos.iglesias@webfoundation.org>                                                          |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |09/09/2015 06:41 AM                                                                                                                               |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Re: [odcgeneralstewardslist] IODC+ DATA ROADMAPS: Gov of Mex on Bundled Commitment on IODC & National Consultation/Roadmap on the Data Revolution |
  |for Sustainable Development                                                                                                                       |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|





Hi all,

I fully agree this is a very important debate and, as Tim mentioned and (as
most of you know) he's been deeply involved with both the CAF and the ODB,
we keep on exploring at WF. The paper he referred to was commissioned to
keeping on exploring the "Use" element.

I was also intrigued about the use of household surveys and met with the
World Justice Project team earlier this year to learn more about their
method. They work with local/regional companies and survey thousands of
people. Besides what Tim mentioned above, two more issues come to mind:
time requirements and cost. Pew's and WPJ's product are not cheap and WPJ's
needs 2 years per round. Complexity of the ODB itself has increased
already. For example, we have introduced this year government
self-assessments as a new data collection component.

I believe we all certainly need to improve how we measure "Use" but also
keeping in mind the perfect might be the enemy of the good.

I'm copying my colleague Carlos as he's currently managing the ODB and our
work on the CAF and may have something else to add.

IIRC, we have organized sessions at the last several conferences on this
topic and we might want to do so again soon, maybe in the context of the
OGP ODWG meeting at the OGP Summit or on the sides of it as I believe most
of us will be there.

Best,
Josema.



2015-09-09 10:12 GMT+02:00 Tim Davies <tim@practicalparticipation.co.uk>:
  This is a really important debate: and if finding good methods for
  assessing levels of open data use would be very valuable.


  Across the components of the Common Assessment Method for Open Data usage
  is the least surveyed - in part due to the complexity of finding good
  robust sampling strategies.

  Surveys: Reflecting on Steve's suggestions around surveys:

  The best examples we probably have of large scale survey work in this
  area is either from the Pew Internet Project, which has a single-country
  US survey capturing American's awareness of Open Government Initiatives (
  http://www.pewinternet.org/2015/04/21/open-government-data/), and then
  the World Justice Project's multi-country survey of major cities, which
  included a number of questions for their Open Government Index (
  http://data.worldjusticeproject.org/opengov/) relating to citizen
  perceptions around their use of Right to Information mechanisms.

  However, particularly when it comes to getting cross-country comparison
  data that is sensitive specifically to open data, as opposed to the
  presence of an app economy or civic technology in general, it can be very
  difficult to frame definitions in surveys in ways that produce reliable
  and comparable data.

  One of the issues faced in the Open Data Barometer's 'Impact' method,
  which broadly combines a measure of use and impact (asking about the
  presence of stories of open data having an impact in particular
  settings), is that countries that spend more resource capturing case
  studies of use may score higher than countries who have more cases of
  use, but where those cases are less well documented or promoted. Re-use
  that doesn't result in high-profile apps and websites is particularly
  likely to be missed by both expert and public-perception surveys.

  From a robust evidence point of view, it's would also be important I
  think to have independent sampling and data collection: making it tricky
  to put govts in the middle of asking citizens to fill out surveys.

  Other approaches: Two other approaches which might be useful here:

  (1) Refining 'data availability' metrics. As Daniel notes, most of our
  measures of data openness right now are not sensitive enough to data
  quality.

  There is some interesting work on domain-specific measures of quality
  (e.g. Open Data Watch Inventory - capturing levels of disaggregation in
  nationals stats:
  http://www.opendatawatch.com/Pages/Open-Data-Inventory.aspx), and finding
  metrics that indicate how re-usable a dataset is likely to be (
  http://www.opendataresearch.org/dl/symposium2015/odrs2015-paper60.pdf).

  I've been interested in exploring whether we can find efficient methods
  for use-case driven testing of the practical openness of datasets to
  replace/complement the current check-list approaches used in the Open
  Data Barometer and Index.

  (2) Finding and evidencing good proxy variables.

  The Open Data Barometer includes variables on civil society capacity, and
  private sector ICT capacity, in part because it hypothesises that these
  are important ingredients of enabling re-use.

  It would be worth testing this in a number of contexts, and exploring
  whether there are other better proxy variables to capture factors aside
  from data quality which are strongly associated with the presence of open
  data re-use in a country.

  ---


  All the best

  Tim






  On Wed, Sep 9, 2015 at 4:07 AM, Steven Adler <adler1@us.ibm.com> wrote:
   Great.  The normal way to measure utilization is through consumer
   preference, but that requires a menu of different consumer choices with
   price discovery.  We have a free commodity with few market alternatives,
   therefore our only option is to survey consumer opinions of open data
   quality, relevance, and value.

   Not very sophisticated but it works if we can develop a short survey and
   get governments to ask users to fill it out anonymously to generate
   reasonable sample sizes.

   What do people think about this?

   Best Regards,


   Steve Adler
   IBM


   Daniel Dietrich --- Re: [odcgeneralstewardslist] IODC+ DATA ROADMAPS:
   Gov of Mex on Bundled Commitment on IODC & National Consultation/Roadmap
   on the Data Revolution for Sustainable Development ---
                                                                           
 From:  "Daniel Dietrich" <daniel.dietrich@okfn.org>                       
                                                                           
 To:    "Steven Adler" <adler1@us.ibm.com>                                 
                                                                           
 Cc:    "Barbara Ubaldi" <Barbara.UBALDI@oecd.org>, "Jose Manuel Alonso" < 
        josema@webfoundation.org>, "Marcio Vasconcelos" <                  
        Marcio.Vasconcelos@avina.net>, "                                   
        ODCstewardslist@opendatacharter.net" <                             
        odcstewardslist@opendatacharter.net>, "sumandro" <                 
        sumandro@cis-india.org>, "" <ZeitzPS@state.gov>                    
                                                                           
 Date:  Tue, Sep 8, 2015 6:08 PM                                           
                                                                           
 Subjec Re: [odcgeneralstewardslist] IODC+ DATA ROADMAPS: Gov of Mex on    
 t:     Bundled Commitment on IODC & National Consultation/Roadmap on the  
        Data Revolution for Sustainable Development                        
                                                                           



   This is a great point! At Open Knowledge we have thought about (but not
   yet found an answer) on how to add the user perspective to the Open Data
   Index, as we have found that some countries actually score relatively
   high in the OD index and OD barometer, but when you go and ask potential
   re-users in those countries they will tell you that the data published
   is actually useless for their work, as its lacks quality (including but
   not limited to: high level of aggregation, missing details
   (itemisation), low granularity, not timely, not updated, no historic
   data for comparison, etc). However it is very hard to capture this kind
   of feedback into an index, as these are individual statements for
   individual use-cases. However not having this perspective the actual
   indexes sometimes draw a misleading picture for some countries.
   Interested to hear other people thoughts. All best Daniel -- Daniel
   Dietrich Co-founder & Chairman Open Knowledge Foundation Germany
   www.okfn.de | info@okfn.de | @okfde Office:



--
You received this message because you are subscribed to the Google Groups
"ODC Stewards list" group.
Visit this group at
http://groups.google.com/a/opendatacharter.net/group/odcstewardslist/.


To unsubscribe from this group and stop receiving emails from it, send an
email to odcstewardslist+unsubscribe@opendatacharter.net.