Re: The role of the AI KR Strategist:

We will be addressing Evaluation in the discussions around use of (StratMl
part 2) KPIs which are essential for measuring success of a strategy.

Agreed - KR activities can start only after the goal and scope of the AI
system have been defined


   - Document the vision, values, goals, objectives for one or more *AIKR
   objects*
   - Employ ontological statements when explaining *AIKR object* audit
   data, veracity facts and (human, social and technology) risk mitigation



It was a pleasure to clarify


On Thu, Jun 11, 2020 at 8:46 PM Paola Di Maio <paola.dimaio@gmail.com>
wrote:

>
> However, we can develop KR say ontologies, which are implementation
> independent,
> Perhaps look at the role of the ontologist, to get a sense of the role for
> the AI KR strategist?
>
> I stress that I dont think of this as a strategy but adopting this
> construct as it is part of the stratml vocab and can be useful way of of
> looking at it
> When it comes to ML, as per your table (ML Evaluation) it is definitely
> not the role of the AI KR strategist to evaluate machine learning imho
> Track (monitor) and evaluate are part of the same task imho
>
> It could be the role of the AI strategist to decide whether ML is needed
> in the AI, and to evaluate the alternatives
> The role of the AI KR strategies would be to devise and implement the KR
> according to the type of AI design/methods being used by the system.
>
> The evaluation of a ML techniques irequires different approaches from the
> evaluation of NL systems, for example
>
> On Fri, Jun 12, 2020 at 8:23 AM Paola Di Maio <paola.dimaio@gmail.com>
> wrote:
>
>> Thank you for clarifying. The table is neat and a good starting point for
>> elaboration. It is a bit scattered but that can be sorted.
>>
>>  I dont see KR  though
>>
>> 1. Can you please point out which of these items fall under KR? (say,
>> ontology?)
>>
>> my experience, the AI KR development work starts after the case for the
>> AI system has already been  made, , ie, when the need for the AI system has
>> already been identified, ( the first two points).  The system design is in
>> place schedules and resources to be in place,
>> KR activities are not AI general system development activities.  What I
>> mean is that KR activities can start only after the goal and scope of the
>> AI system have been defined.  (not sure whose role is to define them,
>> perhaps the system developer or the AI strategist)
>>
>> On Thu, Jun 11, 2020 at 10:44 PM carl mattocks <carlmattocks@gmail.com>
>> wrote:
>>
>>>
>>> Please peruse this GOAL Dependency approach to sequencing
>>>
>>> *Goal Activities*
>>>
>>> *Title AI Strategist Goal*
>>>
>>> *Goal Dependency*
>>>
>>> *AIKR Strategist Protocol*
>>>
>>> Give an understanding of the possible applications of AI to
>>> conversations/decisions about business strategy
>>>
>>>
>>>
>>> Business Strategy
>>>
>>>
>>>
>>> Document, Requirements, Quality, Robust, Ontological Statements, Ethics,
>>> Lawful, Machine Learning Evaluation, Track
>>>
>>> yes
>>>
>>> Identify which areas of the requirements warrant AI solutions versus
>>> which can be achieved with other types of solutions
>>>
>>>
>>>
>>> Requirements
>>>
>>>
>>>
>>> Business Strategy, Document, Quality, Robust, Ontological Statements,
>>> Ethics, Lawful, Machine Learning Evaluation, Track
>>>
>>> yes
>>>
>>> Document the vision, values, goals, objectives for one or more AIKR
>>> objects
>>>
>>>
>>>
>>> Document
>>>
>>>
>>>
>>> Business Strategy, Requirements, Quality, Robust, Ontological
>>> Statements, Ethics, Lawful, Machine Learning Evaluation, Track
>>>
>>> yes
>>>
>>> Define the limits of quality. If a product has limits of
>>> quality/action, then these should be stated
>>>
>>> Quality
>>>
>>>
>>>
>>> Business Strategy, Document, Requirements, Robust, Ontological
>>> Statements, Ethics, Lawful, Machine Learning Evaluation, Track
>>>
>>> yes
>>>
>>> Evaluate machine learning models
>>>
>>>
>>>
>>> Machine Learning Evaluation
>>>
>>>
>>>
>>> Business Strategy, Document, Requirements, Quality, Robust, Ontological
>>> Statements, Ethics, Lawful, Track
>>>
>>> yes
>>>
>>> Track AIKR object performance outcome via KPI (Key Performance
>>> Indicator) based on supervised learning models measurements
>>>
>>>
>>>
>>> Track
>>>
>>>
>>>
>>> Document, Requirements, Quality, Robust, Ontological Statements,
>>>  Machine Learning Evaluation
>>>
>>> yes
>>>
>>> Ensure AI Systems are designed to handle uncertainty and tolerate
>>> perturbation from a likely threat perspective, such as, design
>>> considerations incorporate human, social and technology risk factors
>>>
>>>
>>>
>>> Robust
>>>
>>>
>>>
>>> Document, Requirements, Quality, Ontological Statements, Ethics, Lawful,
>>> Machine Learning Evaluation, Track
>>>
>>> yes
>>>
>>> Employ ontological statements when explaining AIKR object audit data,
>>> veracity facts and (human, social and technology) risk mitigation
>>>
>>> Ontological Statements
>>>
>>>
>>>
>>> Business Strategy, Document, Requirements, Quality, Robust, Ethics,
>>> Lawful, Machine Learning Evaluation, Track
>>>
>>> yes
>>>
>>> Ensure AI Systems adhere to principles of ethics
>>>
>>>
>>>
>>> Ethics
>>>
>>>
>>>
>>> Document, Requirements, Quality, Robust, Ontological Statements, Lawful,
>>> Machine Learning Evaluation, Track
>>>
>>> yes
>>>
>>> Ensure AI Systems comply with all applicable laws and regulations, such
>>> as, provision audit data defined by a governance operating model
>>>
>>>
>>>
>>> Lawful
>>>
>>>
>>>
>>> Document, Requirements, Quality, Robust, Ontological Statements, Ethics,
>>> Machine Learning Evaluation, Track
>>>
>>> yes
>>>
>>> Carl Mattocks
>>>
>>> It was a pleasure to clarify
>>>
>>>
>>> On Wed, Jun 10, 2020 at 8:46 AM carl mattocks <carlmattocks@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Please find attached (summary extract below)  Strategic Plan - The role
>>>> of the AI KR Strategist (A strategist is a person with responsibility
>>>> for the formulation and implementation of a strategy).
>>>>
>>>> To help validate the list of goals we invite proposals for a sequencing
>>>> (of the goals) that could be the foundation of an AI Strategy Protocol.
>>>>
>>>> Please provide your proposals as a reply to this post.. intent is to
>>>> apply them to the StratMl version at 9:00 am (NY time) June 23 meeting.
>>>>
>>>>
>>>>
>>>> thanks
>>>>
>>>> Carl Mattocks
>>>> AIKRCG co-Chair
>>>>
>>>>
>>>> Strategic Plan
>>>>
>>>> *The role of the AI KR Strategist*
>>>>
>>>> For: *Artificial Intelligence Knowledge Representation Community Group
>>>> (AIKR CG)*
>>>>
>>>> Submitted By: *Carl Mattocks*
>>>>
>>>> y *CarlMattocks@WellnessIntelligence.Institute*
>>>>
>>>> *Overview*
>>>>
>>>> This plan defines the role of the AI KR Strategist.
>>>>
>>>> Contents
>>>>
>>>> *No table of contents entries found.*
>>>> 1.    Articulation 1.1.    Vision
>>>>
>>>> For all AI systems to have clearly and transparently documented goals
>>>> and performance data showing that they are being achieved.
>>>> 1.2.    Mission
>>>>
>>>> The mission of an AI Strategist is to define the purpose and goals of
>>>> AI systems, as well as the KPIs by which we can determine if the system is
>>>> meeting its goals.
>>>> 1.3.    Scorecard
>>>>
>>>>
>>>>
>>>> *Goals*
>>>>
>>>> *Objectives*
>>>>
>>>> *Performance Indicators*
>>>>
>>>> *Commentary*
>>>>
>>>> *Goals with no perspective*
>>>>
>>>> Quality (see pp1)
>>>>
>>>> Adherence to Environmental Impacts (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Efficiency (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Ethics (see pp1)
>>>>
>>>> Accountability (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Autonomy (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Confidentiality (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Veracity (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Machine Learning Evaluation (see pp1)
>>>>
>>>> Track (see pp1)
>>>>
>>>> Precision Recall
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Accuracy
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Confusion Matrix
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Per-class accuracy
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Log-Loss
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> AUC-ROC Curve
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> F-measure
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> NDCG
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regression Analysis
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Quantiles of Errors
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> "Almost correct" predictions
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Trustworthy (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Lawful (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Ontological Statements (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Track (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Business Strategy (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Requirements (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Document (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Robust (see pp1)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 1.4.    Goals....
>>>>
>>>> It was a pleasure to clarify
>>>>
>>>>
>>>> On Tue, Jun 9, 2020 at 11:56 PM Paola Di Maio <paola.dimaio@gmail.com>
>>>> wrote:
>>>>
>>>>> oops-
>>>>> Thanks Carl Owen and Chris
>>>>> for capturing and sharing.
>>>>> I repeat here my comment shared with participants after the session -
>>>>> also pasted below
>>>>>
>>>>> if the plan we worked on yesterday is to describe the role of an AI
>>>>> Strategist
>>>>> then we need to make explicit what the goal of the role and associated
>>>>> activities  is/are
>>>>>
>>>>> AI systems /software development can be very sophisticated and well
>>>>> understood practice
>>>>> in systems/software engineering . It is not a strategy role to do
>>>>> requirements analysis and write
>>>>> technical systems specifications, especially not technically advanced
>>>>> ones
>>>>>  So we should review and devise an AI strategist/strategy goals
>>>>> accordingly, taking into account the technical system specifications.
>>>>> or viceversa?
>>>>> A strategist could identify the need for a technical system based on a
>>>>> business problem/need, but then pass the development
>>>>> to a specialist (a systems developer/engineer) who has the skills to
>>>>> do so
>>>>> So upon further reflection, I dont think requirements belong to an ai
>>>>> strategy. unless better specified
>>>>>
>>>>> pdm
>>>>> paola wrote:
>>>>> Thanks for starting this plan
>>>>> As I read what had been drafted I commented with my software/system
>>>>> developer hat on
>>>>> The system lifecycle is considered a process, rather than a strategy,
>>>>> and consists of
>>>>> identifying requirements, specifications and documentation and all
>>>>> Its well understood and documented and there are several models, but
>>>>> the lifecycle itself
>>>>> is fairly standard, it applies to AI systems as to any system
>>>>> So before I have a go at reorganising and perhaps streamlining a bit
>>>>> those goals
>>>>> I think we should define better what is the role of the AI strategist
>>>>> does it overlap the role of a ai system analyst and designer (which is
>>>>> what I have in mind)
>>>>> ie  identify what needs to be done
>>>>> or is it more a role to decide HOW these goals need to be achieved in
>>>>> terms of resources, schedules, policies, coordination with management
>>>>> etc
>>>>> Also, as I mentioned, this may depend on what other roles the AI
>>>>> strategist works with/depends on-
>>>>> p
>>>>>
>>>>>
>>>>> On Wed, Jun 10, 2020 at 11:45 AM Paola Di Maio <paola.dimaio@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 10, 2020 at 12:38 AM carl mattocks <
>>>>>> carlmattocks@gmail.com> wrote:
>>>>>>
>>>>>>> Role of Data Scientist has a process that is described in the links
>>>>>>> below :
>>>>>>>
>>>>>>> *Cross-industry standard process for data mining*, known as
>>>>>>> *CRISP-DM*,[1]
>>>>>>> <https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining#cite_note-Shearer00-1> is
>>>>>>> an open standard <https://en.wikipedia.org/wiki/Open_standard> process
>>>>>>> model that describes common approaches used by data mining
>>>>>>> <https://en.wikipedia.org/wiki/Data_mining> experts. It is the most
>>>>>>> widely-used analytics <https://en.wikipedia.org/wiki/Analytics>
>>>>>>>  model
>>>>>>>
>>>>>>> https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining
>>>>>>>
>>>>>>>
>>>>>>> these lecture notes provide good background notes
>>>>>>>
>>>>>>>
>>>>>>> https://wiwi.hs-duesseldorf.de/personen/thomas.zeutschler/Documents/HSD_W_ITAiBA_Zeutschler_SS2016_Lecture2_CRSIP_DM.pdf
>>>>>>>
>>>>>>>
>>>>>>> this is an IBM presentation on the  The Data Science Process
>>>>>>>
>>>>>>> https://www-01.ibm.com/events/wwe/grp/grp304.nsf/vLookupPDFs/Polong%20Lin%20Presentation/$file/Polong%20Lin%20Presentation.pdf
>>>>>>>
>>>>>>>
>>>>>>> enjoy
>>>>>>> Carl
>>>>>>>
>>>>>>> It was a pleasure to clarify
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 9, 2020 at 12:13 PM Owen Ambur <Owen.Ambur@verizon.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Carl, thanks for leading another productive editing session on the
>>>>>>>> AIKR CG televideo conference today.  I found myself agreeing with much of
>>>>>>>> what Paola was saying, and I continue to be impressed by Chris' capability
>>>>>>>> to comprehend and capture the essence of our dialog in his StratNavApp.
>>>>>>>> For the benefit of those who were unable to join us, I am providing a link
>>>>>>>> here
>>>>>>>> <https://www.stratnavapp.com/StratML/Part1/861566c8-e9be-4642-b52f-f673fa499f4e/Styled>
>>>>>>>> so that they can view our draft plan as it currently exists.
>>>>>>>>
>>>>>>>> I'm sorry that Justin was not able to participate to ensure that we
>>>>>>>> are capturing the knowledge he has gained in considering the roles of AI
>>>>>>>> strategists.  However, it would be great if he and others could contribute
>>>>>>>> their comments and edits on our draft in Chris' app
>>>>>>>> <https://www.stratnavapp.com/> and perhaps participate in our text
>>>>>>>> televideo conference on June 23.
>>>>>>>> While the StratML collection does not yet contain a model
>>>>>>>> performance plan for Data Scientists, it does include one for Chief Data
>>>>>>>> Officers (CDOs), at https://stratml.us/drybridge/index.htm#PP4CDO
>>>>>>>>
>>>>>>>> Unless someone has a better source, I will soon convert the
>>>>>>>> contents of this one to StratML format:
>>>>>>>> https://www.sas.com/en_us/insights/analytics/what-is-a-data-scientist.html
>>>>>>>>
>>>>>>>> BTW, Carl, your reference to OASIS's Business Centric Methodology
>>>>>>>> (BCM) prompted me to convert its content to StratML format at
>>>>>>>> https://stratml.us/drybridge/index.htm#BCM
>>>>>>>>
>>>>>>>> These elements of the BCM are closely related to the purposes of
>>>>>>>> the StratML standard:
>>>>>>>>
>>>>>>>> Objective 1.1: Goals
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c0b98-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Determine the business goals.
>>>>>>>> Objective 1.3: Participants
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c0d32-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Identify the project participants.
>>>>>>>> Objective 1.4: COIs
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c0e40-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Identify the Community of Interest.
>>>>>>>> Goal 2: Goals
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c1106-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Understand the business goals.
>>>>>>>> Objective 3.3: Standards Bodies
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c1750-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Map interoperability requirements to standard bodies.
>>>>>>>> Objective 3.4: Formats
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c182c-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Map interoperability requirements to internal legacy system formats.
>>>>>>>> Objective 3.5: Templates
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c191c-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Capture information in a way that can be reused over time and among
>>>>>>>> participating organizations.
>>>>>>>> Objective 3.5.1: Strategies
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c19f8-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Implement strategies for interoperability.
>>>>>>>> Objective 3.5.5: Standards
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c1da4-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Apply standards for interoperability and contracts.
>>>>>>>> Objective 3.5.7: Performance
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c206a-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Monitor and manage performance.
>>>>>>>> Objective 4.1: Standards
>>>>>>>> <https://stratml.us/carmel/iso/BCMwStyle.xml#_6b2c2b82-aa65-11ea-9885-42f42983ea00>
>>>>>>>> - Use existing standards.
>>>>>>>>
>>>>>>>> It would be great if BCM practitioners were to use the StratML
>>>>>>>> standard (ISO 17469-1).  Anyone who may wish to apply the BCM template as a
>>>>>>>> performance plan for their organization in StratML Part 2 format could do
>>>>>>>> so by clicking on the link provided here.
>>>>>>>>
>>>>>>>> Owen
>>>>>>>>
>>>>>>>> -------- Forwarded Message --------
>>>>>>>> Subject: Re: ai strategist role
>>>>>>>> Date: Tue, 9 Jun 2020 11:22:53 -0400
>>>>>>>> From: carl mattocks <carlmattocks@gmail.com>
>>>>>>>> <carlmattocks@gmail.com>
>>>>>>>> To: Paola Di Maio <paoladimaio10@googlemail.com>
>>>>>>>> <paoladimaio10@googlemail.com>
>>>>>>>> CC: Owen Ambur <Owen.Ambur@verizon.net> <Owen.Ambur@verizon.net>,
>>>>>>>> Chris Fox <chris@chriscfox.com> <chris@chriscfox.com>, Paul Alagna
>>>>>>>> <pjalagna@gmail.com> <pjalagna@gmail.com>
>>>>>>>>
>>>>>>>> Data Scientist is a complimentary role.. which is supported by an
>>>>>>>> open source method that has strong emphasis on machine learning and is Plan
>>>>>>>> Do Check Act oriented.
>>>>>>>>
>>>>>>>> Carl
>>>>>>>>
>>>>>>>> On Tue, Jun 9, 2020, 11:12 AM Paola Di Maio <paola.dimaio@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for starting this plan
>>>>>>>>> As I read what had been drafted I commented with my
>>>>>>>>> software/system developer hat on
>>>>>>>>> The system lifecycle is considered a process, rather than a
>>>>>>>>> strategy, and consists of
>>>>>>>>> identifying requirements, specifications and documentation and all
>>>>>>>>> Its well understood and documented and there are several models,
>>>>>>>>> but the lifecycle itself
>>>>>>>>> is fairly standard, it applies to AI systems as to any system
>>>>>>>>> So before I have a go at reorganising and perhaps streamlining a
>>>>>>>>> bit those goals
>>>>>>>>> I think we should define better what is the role of the AI
>>>>>>>>> strategist
>>>>>>>>> does it overlap the role of a ai system analyst and designer
>>>>>>>>> (which is what I have in mind)
>>>>>>>>> ie  identify what needs to be done
>>>>>>>>> or is it more a role to decide HOW these goals need to be achieved
>>>>>>>>> in
>>>>>>>>> terms of resources, schedules, policies, coordination with
>>>>>>>>> management etc
>>>>>>>>> Also, as I mentioned, this may depend on what other roles the AI
>>>>>>>>> strategist works with/depends on-
>>>>>>>>> p
>>>>>>>>>
>>>>>>>>