nest for knowledge assisted analysis petr berka uep, praha thanos athanasiadis ntua, athens
TRANSCRIPT
![Page 1: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/1.jpg)
NEST for Knowledge Assisted Analysis
Petr BerkaUEP, Praha
Thanos AthanasiadisNTUA, Athens
![Page 2: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/2.jpg)
Knowledge Assisted Analysis
![Page 3: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/3.jpg)
KAA for Images A set of regions is generated by an
initial segmentation of images MPEG-7 Visual Descriptors (dominant
color, texture, shape) are extracted Spatial relations (left-of, above-of, etc.) Regions Adjacency Graph as image
representation
![Page 4: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/4.jpg)
Regions Adjacency Graph A graph’s node represents a
segment/region, where visual information (MPEG-7 descriptors, spatial relations, region mask, contour, etc.) is stored
A graph’s edge represents link between two regions, holding the overall neighboring information
![Page 5: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/5.jpg)
Region labelingFor each region: Visual Descriptor matching with the
instances of the concepts in the domain ontology
Calculation of a combined distance from multiple descriptors
Assignment of labels (concepts) along with a confidence of value -> fuzzy set of labels
Hierarchical merging of regions based on the fuzzy set of labels
![Page 6: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/6.jpg)
KAA architecture
![Page 7: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/7.jpg)
Semantic based segmentation (1/2)
Approach Graph-based representation of images Semantic vs. Syntactic: regions are
assigned fuzzy set of labels instead of low-level features
Modification of traditional segmentation algorithms to operate on labeled regions
Simultaneous image segmentation and region labeling
![Page 8: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/8.jpg)
Semantic based segmentation (2/2)
Target: Solve over-segmentation problems Assign labels with confidence values to
regions Link labels with concepts existing in
ontologies
![Page 9: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/9.jpg)
KAA example results
![Page 10: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/10.jpg)
NEST …
![Page 11: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/11.jpg)
Expert systems at UEP - history of the NEST
May 2003: begining of implementation (P. Berka, V. Laš) DELPHI under Windows knowlege base represented in XML stand-alone + client/server (web) version knowledge base editor czech and english versions
http://lisp.vse.cz/NEST
![Page 12: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/12.jpg)
Knowledge representation (1/4)
Attributes and propositions binary True, False single nominal values of attribute multiple nominal values of attribute numeric fuzzy intervals
sources and actions related to attributes
attribute describes case or environment
![Page 13: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/13.jpg)
Knowledge representation (2/4)
Rules with priorities
IF condition THEN conclusion AND actionwhere condition is disjunctive form (disjunction of conjunctions) of literals (propositions or their negations), conclusion is a list of literals and action is a list of actions (external programms)
compositional - each literal in conclusion has a weight
apriori - compositional rules without condition logical - non-compositional rules without weights;
only these rules can infer true or false
![Page 14: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/14.jpg)
Knowledge representation (4/4)
Integrity constraints
ANT SUC (degree) where ANT and SUC are DNF of literals and degree is a number expressing the importance of the integrity constraint
used to check logical consistency of the consultation
diagnosis(TBC) not diagnosis(healthy)
Contexts - disjunctive form of literals, that (iff having positive weight) determines the applicability
of a rule or integrity constraint
![Page 15: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/15.jpg)
Inference Inference in the network of rules as a
combination of backward and forward chaining compositional inference for compositional and
apriori rules (combining contributions of rules) non-compositional inference for logical rules
(modus ponens + disjunction)
Evaluation of integrity constraintsIMPL(a,s) = max(0, min(1, a-s)) pro a > 0
![Page 16: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/16.jpg)
Uncertainty processing (1/4)
(Based on the algebraic theory of P. Hájek)
defined combination functions on [-1, 1]: NEG to compute the weight of negation, CONJ to compute the weight of conjunction, DISJ to compute the weight of disjunction, CTR to compute the contribution of a rule to
the weight of conclusion, GLOB to combine contributions of more rules.
![Page 17: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/17.jpg)
Uncertainty processing (3/4)
Inferenčnímechanismus
Funkce CTR(a,w) prováhu předpokladu a>0
Fumkce GLOB(w1 ,…, wk)
standardní a*w (w1+w2) / (1 + w1*w2)logický sign(w)*max(a+|w|-1) min(w>0w, 1) - min(w<0 |w|, 1)neuronový a*w min(max(iwi, -1),1)hybridní standardní logický
NEG(w) = - w CONJ(w1,w2) = min(w1,w2)
DISJ(w1,w2) = max(w1,w2)
![Page 18: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/18.jpg)
Modes of consultation
dialogue mode - classical question/answer mode that selects current question using backward chaining
dialogue/questionnaire mode – user can input some volunteer information (using questionnaire), during furthe consultation the system asks questions if needed
questionnaire mode – after filling in the questionnaire the system directly inferrs the goals using forward chaining
input answers form a file – answers can be changed using questionnaire
![Page 19: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/19.jpg)
Types of answers
binary attribute - weight single nominal attribute – value and weight multiple nominal attribute – list of values and
their weights numeric attribute - value
Questions not answered during consultation get the default answer „unknown“ [-1,1] or “irrelevant“ [0,0],
Answers can be postponed - user can return to them after finishing the consultation
![Page 20: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/20.jpg)
Consultation setup
![Page 21: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/21.jpg)
Input from file (batch mode)
![Page 22: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/22.jpg)
… for KAA
![Page 23: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/23.jpg)
Basic idea (1/2)
Expert system NEST (or it’s principles) can be used in KAA for: re-labeling a region if the original labeling has
low confidence proposing to merge a region with it’s neighbors
These two tasks can be solved separately, by two different knowledge bases (expert systems – ES).
![Page 24: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/24.jpg)
Basic idea (2/2)
Because NEST cannot express relations between objects, NEST will be used to process the image locally, i.e. to process one object in one step. So, NEST will be activated repeatedly for different regions in the image. This will require to determine some control mechanism that will decide:
what region to take when to stop processing
![Page 25: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/25.jpg)
Processed part of image – version 1 (Athens meeting)
![Page 26: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/26.jpg)
NEST for re-labeling (1/5)
Input: labels (and confidences) of the
processed region labels (and confidences) of the
neighboring region some global info?
![Page 27: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/27.jpg)
NEST for re-labeling (2/5)So the input can be for example “sky(0.6), sea(0.8)
…” . To be able to reason about the confidences, NEST has to turn them into (fuzzy) intervals like “very_low”, “low”, “medium”, “high”, “very_high” – this can be easily done when creating the knowledge base:
![Page 28: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/28.jpg)
NEST for re-labeling (3/5) Output:
(new) labels (and confidences) of the processed region
Used knowledge:IF the labels have high confidence, THEN don’t change the labelsELSE change the labels according to the
neighbors
![Page 29: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/29.jpg)
NEST for re-labeling (4/5)
Examples of rules: IF old_label(sky) THEN new_label(sky) (1)
IF region_confidence_sky(very_low) AND region_confidence_sea(very_low) AND region_confidence_sand(very_low) AND west_confidence_sky(high) AND east_confidence_sky(high) THEN new_label(sky) (0.6),new_label(unidentified)(0.9)
![Page 30: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/30.jpg)
NEST for re-labeling (5/5) General strategy:This module can be activated for each region
once e.g. according to the confidence of labeling, starting with lowest confidence. The stopping criterion can have a form of a threshold of the confidence.
![Page 31: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/31.jpg)
NEST for merging (1/3) Input:
labels (and confidences) of the processed region labels (and confidences) of the neighboring region some global info?
Output: recommendation for processed region: e.g.
merge_west, merge_east, merge_north, merge_south, don’t_merge
probably also labels (and confidences) for the merged region
![Page 32: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/32.jpg)
NEST for merging (2/3) Used knowledge:
The knowledge can be generalized as follows:IF the neighbors have same labels THEN merge ELSE don’t merge
Example: THEN don’t_merge (0.5) {apriori rule that says,
that we prefer not to merge} IF region_confidence_sky(high) AND
west_confidence_sky(hig) THEN merge_west (1), merged_confidence(sky) (0.85)
![Page 33: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/33.jpg)
NEST for merging (3/3) General strategy:This module can be activated for the regions
repeatedly (performing a kind of bottom-up clustering of regions), starting e.g. with region with highest visual salience? The stopping criterion can have a form of a threshold of the salience (not to handle uninteresting regions)?
![Page 34: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/34.jpg)
Processed part of image – version 2 (EKAW poster)
![Page 35: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/35.jpg)
NEST for merging (1/3) Input:
labels (and confidences) of the regions A and B
labels (and confidences) of the neighboring regions
some global info? Output:
(dis)similarity between regions A and B
![Page 36: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/36.jpg)
NEST for merging (2/3) Used knowledge:
The knowledge can be generalized as follows:
IF regions A and B have same labels THEN similar
ELSE dissimilar Example:
IF regionA_confidence_sky(high) AND regionB_confidence_sky(high) THEN similar (1)
![Page 37: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/37.jpg)
NEST for merging (3/3) General strategy (Semantic Recursive
Shortest Spanning Tree):1. use NEST to evaluate dissimilarity of
current neighbors (edges in ARG)2. select neighbors with lowest dissimilarity3. IF this value is below given threshold
THENa. merge neighbors A and Bb. assign new label
c. goto step 1.
![Page 38: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/38.jpg)
Alternative approachMerge two adjacent regions if they have
the same distribution (in quantitative sense) of classes:
assign labels according to different combinations of majority classes - so for k classes (like sea, sky …) we will have 2k-1
labels merge neighbor regions with the same label
(Inspired by my algorithms for discretization and grouping)
![Page 39: NEST for Knowledge Assisted Analysis Petr Berka UEP, Praha Thanos Athanasiadis NTUA, Athens](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e495503460f94b3ccf5/html5/thumbnails/39.jpg)
NEST – to do Create knowledge base Implement convertors to transform
data between KAA and NEST (xml -> xml)
Build inference mechanism of NEST into KAA