Re: Atb.: [ACTION-256]: Compile and circulate itsTool examples togehter with proposal text

Hi Marcis,
You are correct, we did not consider in depth this differenatiation when 
first considering the text analysis annotation data category, and we 
can't apply several  text analytics annotations to the same text because 
of complete overides.

The structure you indicate may be a bit complex to parse in cases where 
only one or a subset of the scores are needed. But I also agree that we 
don't want to have to add lots of new data categories to address this.

Another question, as raised in the meeting today, is whether for 
disambiguation the confidence score might be better as a part of the 
disambiguation data categoty? We need to address that, but it doesn't 
help with the need you indicate for expressing confidence for the 
terminology and perhaps domain data categories.

Another approach is to generalise the different confidence scores in a 
single data category along the lines sketched out below, effectively 
merging

I think this would still work OK with its-tool _if_ we can refer to 
individual ITS attributes rather then just complete data categories as 
specified at present.

cheers,
Dave


      Processing Confidence [strawman]


        Definition

The *Processing Confidence *data category is used to communicate the 
self reported confidence of the accuracy of an annotation. It is not 
intended to serve as a comparison metric between different engines 
providing specific annotations, but solely for providing self-reported 
confidence by the specific system that produced the annotations. This 
data category does NOT aim to establish any sort of correlation between 
the self-reported confidence and either human evaluation of  usefulness 
of the output.

This data category can be used for several purposes, including, but not 
limited to:

  * Automated prioritisation of annotations for further processing based
    on a quality threshold.

  * ·Providing readers, translators, post-editors, reviewers and
    proofreaders of machine translated text with self-reported relative
    accuracy prediction.
  * Automated prioritising of raw machine translated text for further
    processing based on empirically set thresholds.
  * Providing readers, translators, post-editors, reviewers and
    proofreaders of machine translated text with self-reported relative
    accuracy prediction.

  * MT confidence scores can be displayed e.g.on websites machine
    translated on the fly, by simple web-based translation editors or
    Computer Aided Translation (CAT) tools.

This data category provides a Confidence Score as a rational number from 
the interval <0;1> or as percentage (0-100%) with up to 2 decimal digits.

The confidence score can be displayed on websites on the fly, by simple 
content management systems. Confidence MAY be displayed for human 
consumers as segment annotation or as color-coded font or background.


        Implementation

For elements, the data category informationinherits 
<http://www.w3.org/International/multilingualweb/lt/drafts/its20/its20.html#def-inheritance>to 
the annotation of textual content of the element,/including/child 
elements, but/excluding/attributes..

LOCAL: At least one of the following:

·A mtConf attribute with a value that is a rational number in the 
interval 0 to 1.

·A disambigConf attribute with a value that is a rational number in the 
interval 0 to 1.

·A domainConf attribute with a value that is a rational number in the 
interval 0 to 1.

·A termConf attribute with a value that is a rational number in the 
interval 0 to 1.





On 01/11/2012 10:02, Mārcis Pinnis wrote:
> Hi everyone,
>
> We had a short discussion with Dave about the different confidence scores and we came up to an understanding that the current version might have issues reflecting various kinds of confidences.
>
> For instance, if we have Named Entity Recognition, Term annotation, then also translation and possible other processes that might affect quality, then:
> 1) The text analysis confidence does not allow distinguishing between multiple confidences for one tagged element (for instance, 0.1 - termAnnotator1; 0.5 termAnnotator2; 0.7 - namedEntityRecogniser1, etc.)
> 2) Having different confidence namings will make the standard quite heavy (do we really know all the types of confidences that people might like to add/use? Maybe we should not restrict to the ones specified?)
>
> One solution that I was thinking could be having something like agentConfidence that may contain a complex structure ("0.1|0.5|0.7|etc.") and then have somewhere specified how to understand this sequence.
>
> Any thoughts? Ideas?
>
> Best regards,
> Mārcis ;o)
> ________________________________________
> No: Dave Lewis [dave.lewis@cs.tcd.ie]
> Nosūtīts: ceturtdiena, 2012. gada 1. novembrī 10:16
> Kam: public-multilingualweb-lt@w3.org
> Tēma: [ACTION-256]: Compile and circulate itsTool examples togehter with  proposal text
>
> Hi,
> The attached file contains a revision to Yves its-tools proposal. I have
> also added revised text for mt confidence score and text analysis
> annotation revised . Finally, I have also added at the end the examples
> from these data categories, showing how the engine identification
> information (which I've now removed from the data category), can be
> repalced using the its-tools element.
>
> There are comments that need to be addressed in the document. We can go
> through these quickly if we wish in the meeting.
>
> cheers,
> Dave
>
>
>
>
>
>