Analysing the ELICIT factoids- example of uncertainty

David Mott (ETS, IBM UK)April 2014

v1.1

David Mott (ETS, IBM UK)April 2014

v1.1

International Technology AllianceIn Network & Information Sciences

International Technology AllianceIn Network & Information Sciences

ELICIT is…

• An experimental framework for researching into collective sensemaking by teams of collaborating humans

– A set of Natural Language sentences providing information about a scenario in a fictitious geographic region with a number of different countries

– There is a possible attack by a group on a target in one of these countries– The collaboration of teams must identify, from the information the who, what, where,

and when of the attack– In different experiments, the teams are given only partial information, and the task is to

communicate and collaborate to do the identification

ELICIT sentences (factoids)1 The Lion is involved2 Word has it that an unprotected target is preferred to ensure the likelihood of success (can assume is true)3 The Lion doesn't operate in Chiland4 The Lion attacks in daylight5 The Azure, Brown, Coral, Violet, or Chartreuse groups may be planning an attack6 The Azure and Violet groups use only their own operatives, never employing locals7 The Chartreuse group is not involved8 The Lion is known to work only with the Azure, Brown, or Violet groups9 The Purple or Gold group may be involved10 All of the members of the Azure group are now in custody11 Reports from the Coral group indicate a reorganization12 There is a lot of activity involving the Violet group13 The Brown group is recruiting locals - intentions unknown14 The Lion will not risk working with locals15 The Jackal has been seen in Tauland16 Members of the Purple group have been visiting Omegaland17 The Chartreuse group has close ties with local media18 The Azure group has a history of attacking embassies19 The Purple and Gold groups have blood ties20 The Brown group has been known to use IED's21 Only the Coral and Violet groups have a capacity to hit protected targets22 All high value targets belonging to Tauland and Epsilonland are well protected23 The attackers are focusing on a high visibility target24 Caches of explosives have recently been found in Epsilonland, Chiland, and Psiland25 Financial institutions in Tauland, Chiland, and Omegaland were recently attacked there is evidence of more attacks26 Reports that uniforms were stolen in Tauland, Epsilonland and Psiland27 Bloggers are discussing the role of financial institutions in oppressing the Coral, Violet and Chartreuse groups28 Members of the Violet and Chartreuse groups were active in planning protests at a recent financial summit29 Security forces are providing highly visible, around the clock protection to all visiting dignitaries in the region30 Dignitaries in Epsilonland employ private guards31 Tau, Epsilon, Chi, Psi and Omega-lands are providing visible, around the clock protection to their own dignitaries at home32 A new train station is being built in the capital of country Tauland33 Tauland's embassy in Epsilonland has a flat roof34 Until recently most of the dignitaries in Tauland rode in Mercedes35 Dignitaries in Chiland have motorcycle escorts36 Epsilonland's embassy in Tauland has two helicopter pads37 The Azure, Brown, Coral, and Violet groups have the capacity to operate in Tau, Epsilon, Chi, Psi and Omega-lands38 Locals in Tauland, Epsilonland and Omegaland are being recruited39 Countries Chiland, Psiland and Omegaland are taking steps to protect their embassies abroad40 The Brown group members have entered Tauland and Epsilonland41 Reports from Tauland, Chiland and Psiland indicate surveillance ongoing at coalition embassies42 The target is a coalition member embassy, visiting dignitary, or financial institution (Tau, Epsilon, Chi, Psi or Omega-lands)43 No traces of members from the Coral group have been found in countries Psiland or Omegaland44 Chiland is in the process of deploying troops to protect the embassies of coalition partners45 The Azure, Brown, and Coral groups want to attack the interests of Tauland, Epsilonland or Chiland46 The Coral and Violet group operatives have entered Psiland

47 All high value targets of Omegaland are well protected48 There has been an increase in messages intercepted in Psiland49 The Lion was born in Tauland50 There is no new information about Brown group operations in Chiland51 Epsilonland is mountainous52 Tauland is land locked53 The attack will be at 11:0054 The Azure and Brown groups prefer to attack at night55 The Tauland embassy in Epsilonland is hosting a international conference on the 10th56 The Chartreuse, Purple and Gold groups are known to attack at any time of the day57 Attacking buildings when there are many people present increases casualties58 The Coral, Chartreuse and Purple groups are capable of attacking year round59 The Lion is planning something in April on the anniversary of his father's death60 There are fewer attacks in the dead of winter (January thru March)61 The Violet and Chartreuse groups want to attach the interests of Chiland, Psiland and Omegaland62 The Violet group is planning something big on the 5th63 The Violet group prefers to operate in daylight64 The lion was born in June65 The Coral group prefers to attack at night66 The Purple group prefers to attack in daylight67 The Brown group needs time to regroup68 The Azure group does not attack on its holy days

WHO (agents involved), WHAT (target of attack),

WHEN, WHERE?

A stretched goal for task 2

• I was told that the sentences were trivially easy to turn into a formal representation which was then a “logic puzzle” to be solved, so why not:

1. Pass the ELICIT sentences through the ERG2. Convert the output to Controlled English, after modelling the domain concepts3. Run a logic puzzle solver that interprets CE4. Solve the problem!5. Thus demonstrating the power of our system to perform analysis in a valid problem, allowing us

to participate in the sensemaking experiments of task 1?

• The sentences ran through the ERG system– Some minor syntactic changes

• X’s Y did not parse, so I changed to the Y of X• …

– Most sentences parsed, though I have not checked all of the parses for reasonableness.

• However there is a big problem…

Ambiguity

• On analysing the sentences formally there are many ambiguities and difficulties of interpretation:

– The Purple group or the Gold group may be involved• What is the meaning of “or” in this context ? Inclusive? exhaustive?

– The attackers are focusing on a high visibility target• What does high visibility mean? Only domain knowledge or common sense can tell

– The Lion will not risk working with locals• Does this mean the Lion will not work with locals?

– Dignitaries in Epsilonland employ private guards• Does “in” mean “belonging to” or “located in”?

– Reports from Tauland, Chiland and Psiland indicate surveillance ongoing at coalition embassies• Are the reports from the embassies or the host countries?

– The Violet group prefers to operate in daylight• Does this exclude artificial light, in this context?• How do we compare this with “at night”

– The Azure and Violet groups use only their own operatives, never employing locals• Actually this is a rule not a fact, (using present tense?) but how do we generalise the statement just

enough to capture the intent?

• I analysed all 68 sentences in some detail as to ambiguities! So even “simple” sentences are difficult to analyse; common sense knowledge is required

Handling uncertainty

• We have:– defined CE domain model– generated CE facts from sentences by

Constructing CE sentences “by hand” This has required “interpretation” of the sentences, introducing

uncertainty factors as assumptions– Run the sentences in the CE system, leading to a

possible solution to the who, what, where and when– Recorded the rationale for the reasoning

Including how the conclusions are based upon assumptions

• This should provide an example:– how the reasoning can be affected by the use of

assumptions

Flow of InformationOriginal

sentences

MRS

WHO

Simplifiedsentences

Domain CEDomain CE

MRS

Domain CEDomain Model

(Concepts, Rules)

WHEN

Problem Solving Strategy

WHAT

WHERE

ERG system

ERG system

CE system

CE system

Manual

Reasoning Engine

Manual

Explanation

This shows the approach in the context of our three different

possible approaches

Uncertainty about protected targets (1)

Word has it that an unprotected target is preferred to ensure the likelihood of success (can assume is true)

Original sentence:

if ( the potential target T is a protected thing )then ( the potential target T is a non-target ).

Actually, because we are taking a constraint-based problem solving

strategy, we would represent this in the opposite negative form

if ( the thing T is a non-protected thing )then ( the thing T is a possible target ).

We could just take this preference as absolutely true, and infer that non-

protected things are possible targets

(The negative form could be derived automatically from the positive form via modus tollens, but this latter reasoning rule

has not been implemented in the CE reasoning engine)

Uncertainty about protected targets (2)

[ rule_prot_targ_non_target ]

if ( the potential target T is a protected thing ) and ( the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target )then( the potential target T is a non-target ).

However the assumption that this is actually true is an “interpretation” of the sentence in the context of the ELICIT domain and may be too strong

it is assumed by the agent dm that the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target.

there is a domain interpretation named prot_targ_non_target thathas "the preference is treated as an absolute truth" as description.

and we ASSUME that this interpretation can be made by the rule as a necessary

premise to infer its conclusion

We wish to explicitly record this interpretation as part of domain

knowledge

Uncertainty about protected targets (3)

[ rule_prot_targ_non_target ]

if ( the potential target T is a protected thing ) and ( the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target )then( the potential target T is a non-target ).

Now any application of the rule will make the conclusions dependent upon the assumption as well as the basic premise propositions

the potential target t1 is a non-target

the potential target t1 is a protected thing

it is assumed by the agent dm that the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target.

Providing this information in the rationale can show the analyst this

source of uncertainty

Uncertainty about security in the region (1)

Security forces are providing highly visible, around the clock protection to all visiting dignitaries in the region

Original sentence:

But how do we interpret “in the region”? We could take the view that all of the

countries mentioned in the ELICIT sentences are in the region we have

called “this region”

[ visiting_digs_protected ]if ( the visiting dignitary D is visiting the country HOSTC ) and ( the country HOSTC is located in the region thisregion )then ( the visiting dignitary D is a protected thing ).

We want to represent the inference that visiting dignitaries in the region are

protected, as in this rule

if ( there is a region named thisregion ) and ( there is a country named C ) andthen ( the country C is located in the region thisregion ).

Uncertainty about security in the region (2)However the assumption that this is actually true is an “interpretation” of the sentence and may be too

strong

it is assumed by the agent dm that the sentence interpretation si29 can be made by the rule assume_allcountries_in_region.

the sentence interpretation si29 has "we take the region to include all countries" as description.

and we ASSUME that this interpretation can be made by the rule as a necessary

premise to infer its conclusion

We wish to explicitly record this interpretation as part of

interpreting the sentence

[ assume_allcountries_in_region ]if ( there is a region named thisregion ) and ( there is a country named C ) and ( the sentence interpretation si29 can be made by the rule assume_allcountries_in_region )then ( the country C is located in the region thisregion ).

Note we have different types of interpretation, those general to the

domain and those specific to a sentence

Identification requires assumptions

• The reasoning that identifies the WHAT and WHERE seems dependent upon these and other assumptions e.g.:

– That protected things are non-targets (domain interpretation)– That all countries are in “this region” (sentence interpretation)– That we know the total set of possible participants and possible targets

(domain interpretation)– That “in” means working for, not located in (sentence interpretation)– That there is only one attack situation being talked about (domain

interpretation)

• We can generate the rationale for the reasoning, and this shows the assumptions used

“Proof tables”

Why cant the Lion work with the Coralgroup?

propositions

rule

• Each column is a rule application

• Red cell is the conclusion• Salmon cells are the premises

• In the second column a green cell indicates a predefined fact and pink represents an assumption and a word represents the name of a rule leading to the conclusion

• The Equivalent Rule in CE

[ no_time_overlap : ] if ( the operative A operates in the time interval TA ) and ( the group B operates in the time interval TB ) and ( the time interval TA does not overlap the time interval TB ) then ( the operative A cannot work with the group B ) .

We will show the rationale in the form of “proof

tables”

Determining the “WHAT”

• In our constraint-based approach we find the WHAT (i.e. the target) by eliminating all other possibilities

• One such possibility is “ChilandDVE” representing the Chiland dignitaries visiting Epsilonland:– there is a dignitary named ChilandDVE that is an official of

the country Chiland and is located in the country Epsilonland.

• We eliminate ChilandDVE by proving:– the possible target ChilandDVE is a non-target.

Reasoning for “Chiland is a non-target”

Assume all countries are in the

regionAssume

protected things are

non-targets

Assumptions used

• It can be seen that the proof of “the possible target ChilandDVE is a non-target” involves several assumptions, based upon an interpretation of the original sentences– Word has it that an unprotected target is

preferred to ensure the likelihood of success (can assume is true)

– Security forces are providing highly visible, around the clock protection to all visiting dignitaries in the region All countries mentioned are in this “region”

Could Argumentation be used?

• Could arguments be had in order to construct the assumed interpretations of the sentences and the domain logic?– The paper on “On Interpreting the ELICIT

sentences” https://www.usukitacs.com/node/2603 has some examples of informal arguments and what the sentences mean

For example:

• For the sentence:– Security forces are providing highly visible, around the clock protection to all

visiting dignitaries in the region• In the initial analysis (done on first exposure to the sentences, with no

preconceived knowledge on the author’s part), it is stated:– This sentence is highly ambiguous, since it does not state the region in

question. It seems unlikely that this refers to all regions, so it is hard to see how this information could be used in inference.

• In subsequent work to formulate the sentences, the view changed to the interpretation that “all countries are in the region”, since the alternative view that “only some are in the region” would not make sense given that:

– there is no information given about the membership of countries in any region

– There is no mention of specific regions, just the idea of “the region”– The sentence would be meaningless if either the initial analysis were true or

there were just “some” countries in the region without stating which– It is unlikely that a sentence is given that is meaningless