Neurocognitive approach Neurocognitive approach to natural language to natural language

understanding and creativityunderstanding and creativity

Włodzisław DuchWłodzisław DuchDepartment of InformaticsDepartment of Informatics, ,

Nicolaus Copernicus UniversityNicolaus Copernicus University, Toruń, Toruń, Poland, Poland

Google: W. DuchGoogle: W. DuchICONIP’08, AucklandICONIP’08, Auckland

PlanPlan

1. Neurocognitive informatics.2. Intuition. 3. Most mysterious thing about the mind … 4. Creativity research: from psychology to neuroscience.5. Words in the brain: creation of novel words. 6. Memory and pair-wise priming.7. Insight.8. Neurocognitive approach to natural language.9. Creation of ideas, mental models.10. What can we understand?

Neurocognitive informaticsNeurocognitive informaticsComputational Intelligence. An International Journal (1984)+ 10 other journals with “Computational Intelligence”,

D. Poole, A. Mackworth R. Goebel, D. Poole, A. Mackworth R. Goebel, Computational Intelligence - A Logical Approach. Computational Intelligence - A Logical Approach. (OUP 1998), GOFAI book, logic and reasoning. (OUP 1998), GOFAI book, logic and reasoning. • CI: lower cognitive functions, perception, signal analysis, CI: lower cognitive functions, perception, signal analysis,

action control, sensorimotor behavior.action control, sensorimotor behavior.• AI: higher cognitive functions, thinking, reasoning, planning etc.AI: higher cognitive functions, thinking, reasoning, planning etc.• Neurocognitive informaticsNeurocognitive informatics:: brain processes can be a great brain processes can be a great

inspiration for AI algorithms, if we could only understand them …. inspiration for AI algorithms, if we could only understand them ….

What are the neurons doing? Perceptrons, basic units in multilayer What are the neurons doing? Perceptrons, basic units in multilayer perceptron networks, use threshold logic – NN inspirations. perceptron networks, use threshold logic – NN inspirations. What are the networks doing? Specific transformations, memory, What are the networks doing? Specific transformations, memory, estimation of similarity.estimation of similarity.How do higher cognitive functions map to the brain activity? How do higher cognitive functions map to the brain activity? Neurocognitive informatics Neurocognitive informatics = abstractions of this process . = abstractions of this process .

IntuitionIntuitionIntuition is a concept difficult to grasp, but commonly believed to play important role in business and other decision making; „knowing without being able to explain how we know”.

Sinclair Ashkanasy (2005): intuition is a „non-sequential information-processing mode, with cognitive & affective elements, resulting in direct knowing without any use of conscious reasoning”. 3 tests measuring intuition: Rational-Experiential Inventory (REI), Myers-Briggs Type Inventory (MBTI) and Accumulated Clues Task (ACT). Different intuition measures are not correlated, showing problems in constructing theoretical concept of intuition. Significant correlations were found between REI intuition scale and some measures of creativity. ANNs evaluate intuitively? Yes, although intuition is used also in reasoning.Intuition in chess has been studied in details (Newell, Simon 1975). Intuition may result from implicit learning of complex similarity-based evaluation that are difficult to express in symbolic (logical) way.

Intuitive thinkingIntuitive thinking

Learning from partial observations: Ohm’s law V=I×R; Kirhoff’s V=V1+V2.

Geometric representation of facts:+ increasing, 0 constant, - decreasing.

True (I-,V-,R0), (I+,V+,R0), false (I+,V-,R0).

5 laws: 3 Ohm’s 2 Kirhoff’s laws.All laws A=B+C, A=B×C , A-1=B-1+C-1, have identical geometric interpretation!

13 true, 14 false facts; simple P-space, but complex neurodynamics.

Question in qualitative physics (PDP book): Question in qualitative physics (PDP book): if if RR22 increases, increases, RR11 and and VVtt are constant, what are constant, what will happen with current and will happen with current and VV11, , VV22 ??

Intuitive reasoningIntuitive reasoning5 laws are simultaneously fulfilled, all have the same representation:

Question: If Question: If RR22=+=+, , RR11=0=0 and and V V =0, what can be said about =0, what can be said about II, , VV11, , VV22 ??Find missing value giving Find missing value giving FF((VV=0, =0, RR, , II,,VV11, , VV22, , RR11=0, =0, RR22=+) >0=+) >0 Assume that one of the variable takes value Assume that one of the variable takes value XX = + = +, is it possible? , is it possible? Not if Not if FF((VV=0, =0, RR, , II,,VV11, , VV22, , RR11=0, =0, RR22=+) =0=+) =0, i.e. one law is not fulfilled. , i.e. one law is not fulfilled. If nothing is known 111 consistent combinations out of 2187 (5%) exist. If nothing is known 111 consistent combinations out of 2187 (5%) exist.

5

1 2 1 21

( , , , , , , ) ( , , )t i i i ii

F V R I V V R R F A B C

Intuitive reasoning, without Intuitive reasoning, without manipulation of symbols. manipulation of symbols. Heuristics: select variable giving Heuristics: select variable giving unique answer, like unique answer, like RRtt. . Soft constraints or semi-quantitative Soft constraints or semi-quantitative => small => small |F(X)| |F(X)| values. values.

Mysterious mind Mysterious mind … … Intuition is relatively easy … what features of our brain/minds are most mysterious? Consciousness? Imagination? Emotions, feelings? Thinking?

Masao Ito (director of RIKEN, neuroscientist) answered: creativity. MIT Encyclopedia of Cognitive Sciences (2001) has 1100 pages. 6 chapters about logics over 100 references to logics in the index. Creativity: 1 page (+1 page about „creative person”). Intuition: 0, not even mentioned in the index.

In everyday life we use intuition and creativity more often than logics.The subject is getting popular … The subject is getting popular …

• Kenneth M. Heilman, Creativity and the Brain, Psychology Press Kenneth M. Heilman, Creativity and the Brain, Psychology Press 20052005• Mario Tokoro Ken Mogi (Sony Labs), Creativity and the Brain, 2007.• Duch W, Creativity and the Brain, W: A Handbook of Creativity for Teachers. Ed. Ai-Girl Tan, Singapore: World Scientific 2007, pp. 507-530

How to define creativity?How to define creativity?Bink Marsh (2001): the number of definitions of „creativity” is equal to the number of researchers that study this subject.Sternberg (ed. Handbook of Human Creativity, 1998): „the capacity to create a solution that is both novel and appropriate”, not only in creation of novel theories or inventions, but also in our everyday actions, language understanding, interactions. Encyclopedia of creativity (Elsevier, 2005), eds. M. Runco S. Pritzke, 167 articles, but no testable models of creativity have been proposed. Journals: Creativity Research Journal, from 1988, LEA.Journal of Creative Behavior, from 1967, Creative Education Foundation.

Many connections with research in: general intelligence, IQ tests, genius, special gifts, idiot savant syndrome and psychopathologies, intuition, insight (Eureka or Aha!), discovery ...

Psychology of Psychology of creativitycreativity

G. Wallas, The art of thought (1926): four-stage Gestalt model of problem solving. 4 stages: preparation, incubation, illumination and verification.Stages identified in creative problem solving by individuals and small groups of people; additional stages may be added: finding or noticing a problem, proposing interesting questions, frustration period preceding illumination, communication following verification etc. Understanding details of such stages and sequences yielding creative productions is a central issue for creativity research, but is it sufficient? Poincare (1948): math intuition and creativity is a discrimination between promising and useless ideas and their combinations; math thinking may be based on heuristic search among sufficiently rich representations. Math intuition is an interplay between spatial imagination, abstraction and approximate reasoning, and analytical reasoning or visual-spatial and linguistic thinking. This is observed in fMRI imaging (S. Dehaene, 1999).

Symbols in the brainSymbols in the brainOrganization of the word recognition circuits in the left temporal lobehas been elucidated using fMRI experiments (Cohen et al. 2004).How do words that we hear, see or are thinking of, activate the brain? Seeing words: orthography, phonology, articulation, semantics.

Lateral inferotemporal multimodal area (LIMA) reacts to auditory visual Lateral inferotemporal multimodal area (LIMA) reacts to auditory visual stimulation, has cross-modal phonemic and lexical links. Adjacent visual stimulation, has cross-modal phonemic and lexical links. Adjacent visual word form area (VWFA) in the left occipitotemporal sulcus is unimodal. word form area (VWFA) in the left occipitotemporal sulcus is unimodal.

Likely: homolog of the VWFA in the auditory stream, the auditory word form Likely: homolog of the VWFA in the auditory stream, the auditory word form area, located in the left anterior superior temporal sulcus. area, located in the left anterior superior temporal sulcus.

Large variability in location of these regions in individual brains.Large variability in location of these regions in individual brains.

Left hemisphere: precise representations of symbols, including Left hemisphere: precise representations of symbols, including phonological components; right hemisphere? Sees clusters of concepts. phonological components; right hemisphere? Sees clusters of concepts.

Words in the brainWords in the brainPsycholinguistic experiments show that most likely categorical, phonological representations are used, not the acoustic input.Acoustic signal => phoneme => words => semantic concepts.Phonological processing precedes semantic by 90 ms (from N200 ERPs).F. Pulvermuller (2003) The Neuroscience of Language. On Brain Circuits of Words and Serial Order. Cambridge University Press.

Phonological neighborhood density = the number of words that are similar in sound to a target word. Similar = similar pattern of brain activations.Semantic neighborhood density = the number of words that are similar in meaning to a target word.

Action-perception networks inferred from ERP and fMRI

Neuroimaging wordsNeuroimaging wordsPredicting Human Brain Activity Associated with the Meanings of Nouns," T. M. Mitchell et al, Science, 320, 1191 (May 30), 2008

•Clear differences between fMRI brain activity when people read and think about different nouns.•Reading words and seeing the drawing invokes similar brain activations, presumably reflecting semantics of concepts.•Although individual variance is significant similar activations are found in brains of different people, a classifier may still be trained on pooled data. •Model trained on ~10 fMRI scans + very large corpus (1012) predicts brain activity for over 100 nouns for which fMRI has been done.

Overlaps between activation of the brain for different words may serve as expansion coefficients for word-activation basis set.Example for verbs.

Semantic => vector repsSemantic => vector repsWord w in the context: (w,Cont), distribution of brain activations.States (w,Cont) lexicographical meanings: clusterize (w,Cont) for all contexts, define prototypes (wk,Cont) for different meanings wk.

Simplification: use spreading activation in semantic networks to define . How does the activation flow? Try this algorithm on collection of texts:

• Perform text pre-processing steps: stemming, stop-list, spell-Perform text pre-processing steps: stemming, stop-list, spell-checking ...checking ...

• Use MetaMap with a very restrictive settings to discover Use MetaMap with a very restrictive settings to discover concepts, avoiding highly ambiguous results when mapping concepts, avoiding highly ambiguous results when mapping text to UMLS ontology. text to UMLS ontology.

• Use UMLS relations to create first-order cosets (terms + all new Use UMLS relations to create first-order cosets (terms + all new terms from included relations); add only those types of terms from included relations); add only those types of relations that lead to improvement of classification results.relations that lead to improvement of classification results.

• Reduce dimensionality of the first-order coset space, leave all Reduce dimensionality of the first-order coset space, leave all original features; use feature ranking method for this reduction. original features; use feature ranking method for this reduction.

• Repeat last two steps iteratively to create second- and higher-Repeat last two steps iteratively to create second- and higher-order enhanced spaces, first expanding, then shrinking the order enhanced spaces, first expanding, then shrinking the space. space.

Create Create XX vectors representing concepts.vectors representing concepts.

Computational creativityComputational creativityGo to the lower level … construct words from combinations of phonemes, pay attention to morphemes, flexion etc.

Start from keywords priming phonological representations in the auditory cortex; spread the activation to concepts that are strongly related.Use inhibition in the winner-takes-most to avoid false associations.Find fragments that are highly probable, estimate phonological probability.Combine them, search for good morphemes, estimate semantic probability.

Creativity = space + imagination (fluctuations) + filtering (competition)

Space: neural tissue providing space for infinite # of activation patterns. Imagination: many chains of phonemes activate in parallel both words and non-words reps, depending on the strength of synaptic connections. Filtering: associations, emotions, phonological/semantic density.

Autoassociative networksAutoassociative networksSimplest networks: • binary correlation matrix, • probabilistic p(ai,bj|w)

Major issue: rep. of symbols,morphemes, phonology …

W

x 0 00 x 00 0 x

x x xx x xx x x

x x xx x xx x x

x 0 00 x 00 0 x

x x xx x xx x x

x x xx x xx x x

x 0 00 x 00 0 x

Words: experimentsWords: experimentsA real letter from a friend: I am looking for a word that would capture the following qualities: portal to new worlds of imagination and creativity, a place where visitors embark on a journey discovering their inner selves, awakening the Peter Pan within. A place where we can travel through time and space (from the origin to the future and back), so, its about time, about space, infinite possibilities.FAST!!! I need it sooooooooooooooooooooooon.

creativital, creatival (creativity, portal), used in creatival.comcreativery (creativity, discovery), creativery.com (strategy+creativity)discoverity = {disc, disco, discover, verity} (discovery, creativity, verity)digventure ={dig, digital, venture, adventure} still new! imativity (imagination, creativity); infinitime (infinitive, time) infinition (infinitive, imagination), already a company nameportravel (portal, travel); sportal (space, sport, portal), taken timagination (time, imagination); timativity (time, creativity)tivery (time, discovery); trime (travel, time) Mambo server at: http://www-users.mat.uni.torun.pl/~macias/mambo

More experimentsMore experiments• Probabilistic model, rather complex, including various linguistic

peculiarities; includes priming.

Search for good name for electronic book reader (Kindle?):

Priming set (After some stemming):• Acquir, collect, gather , air, light, lighter, lightest, paper, pocket,

portable, anyplace, anytime, anywhere, cable, detach, global, globe, go, went, gone, going, goes, goer, journey, move, moving, network, remote, road\$, roads\$, travel, wire, world, book, data, informati, knowledge, librar, memor, news, word, words, comfort, easi, easy, gentl, human, natural, personal, computer, electronic, discover, educat, learn, read, reads, reading, explor.

Exclusion list (for inhibition): • aird, airin, airs, bookie, collectic, collectiv, globali, globed, papere,

papering, pocketf, travelog.

More wordsMore wordsCreated word Word count and # domains in Google• librazone 968 1 • inforizine -- -- • librable 188 -- • bookists 216 -- • inforld 30 -- • newsests 3 -- • memorld 78 1 • goinews 31 -- • libravel 972 -- • rearnews 8 -- • booktion 49 -- • newravel 7 -- • lighbooks 1 -- + popular infooks, inforion, datnews, infonews, journics

Memory & creativityMemory & creativityCreative brains accept more incoming stimuli from the surrounding environment (Carson 2003), with low levels of latent inhibition responsible for filtering stimuli that were irrelevant in the past. “Zen mind, beginners mind” (S. Suzuki) – learn to avoid habituation! Complex representation of objects and situations kept in creative minds.

Pair-wise word association technique may be used to probe if a connection between different configurations representing concepts in the brain exists. Conclusions from priming experiments: is there connection between two words? Semantic relations between [milk, cow] is obvious, but between [chocolate, squirrel] is not. What if additional “mask” word is added in between? Depends whether mask is related in phonological or semantic sense, or just random letters. In creative brains: connections are denser, “depth of processing” is deeper, difficult associations are easier to find, but recognition=formation of attractor states, may take longer if neutral masking is used.

From words to ideasFrom words to ideasIs creativity based on unconstrained imagination, no rules?

No! Anarchist unstructured approaches fail (free associations, brainstorming, random stimulation or lateral thinking)! Structured approaches, based on higher-order rules and templates, are better. Goldenberg, Mazursky & Solomon, Science 285, 1999.J. Goldenberg D. Mazursky, Creativity in Product Innovation, CUP 2002Avoid problems with complex knowledge reps using associative memory!

Replacement schema for advertising of product P: 1. Link products P to relevant traits T (270 were collected from adds). Ex: car – speed, economy, reliability, comfort ... 2. List symbols S that always invoke desired trait T. Ex: speed=light, bullet . For each trait 3-4 most frequent symbols were selected.3. Link symbols S with objects A that exemplifies them. 4. Link products P with objects A => generates idea for advertisement.

Replacement schemeReplacement schemeTask: create advertisement for Nike air shoes.

Product P = Nike air shoesTrait T: “cushioning and absorbing the shocks” caused by jumping.Symbol S that invokes T: life net for fire victims jumping from a burning building.Replace S with P.Result: propose advertisement:

firemen holding a giant shoe!Ideas generated by the automated routine were presented to judges, along with ideas on the same theme appearing in magazine ads and advertising ideas generated by layman individuals.

Magazine ads: 2.880.55, templates 2.890.48, laymens 2.220.43 Winning adds: 3.26 0.49

Problems requiring Problems requiring insightsinsights

Given 31 dominos and a chessboard with 2 cornersremoved, can you cover all board with dominos?

Analytical solution: try all combinations.

Does not work … too many combinations to try.Logical, symbolic approach has Logical, symbolic approach has little chance to create proper little chance to create proper activations in the brain, linking activations in the brain, linking new ideas: otherwise there will new ideas: otherwise there will be too many associations, be too many associations, making thinking difficult. making thinking difficult.

Insight <= right hemisphere, Insight <= right hemisphere, meta-level representations meta-level representations without phonological (symbolic) without phonological (symbolic) components ... counting? components ... counting?

dd oo

mm ii

nnoo

phonological repsphonological reps

chess boardchess board

blackblackwhitewhite

dominodomino

Insights and brainsInsights and brainsActivity of the brain while solving problems that required insight and that could be solved in schematic, sequential way has been investigated. E.M. Bowden, M. Jung-Beeman, J. Fleck, J. Kounios, „New approaches to demystifying insight”. Trends in Cognitive Science 2005.After solving a problem presented in a verbal way subjects indicated themselves whether they had an insight or not.

An increased activity of the right hemisphere anterior superior temporal An increased activity of the right hemisphere anterior superior temporal gyrus (RH-aSTG) was observed during initial solving efforts and insights. gyrus (RH-aSTG) was observed during initial solving efforts and insights. About 300 ms before insight a burst of gamma activity was observed, About 300 ms before insight a burst of gamma activity was observed, interpreted by the authors as interpreted by the authors as „„making connections across distantly related making connections across distantly related information during comprehension ... that allow them to see connections information during comprehension ... that allow them to see connections that previously eluded themthat previously eluded them”.”.

Insight interpretedInsight interpretedWhat really happens? My interpretation:

• LH-STG represents concepts, S=Start, F=final• understanding, solving = transition patterns, step by step, from S to F• if no connection (transition) is found in LH this leads to an impasse; • RH-STG observes LH activity on meta-level, clusters concepts into

abstract categories (cosets, or constrained sets), has no linguistic reps;

• connection between S and F is found in RH at non-symbolic level, leading to a feeling of vague understanding;

• gamma burst increases the activity of LH representations for S, F and intermediate configurations; feeling of imminent solution arises;

• stepwise transition between S and F is found;• finding solution is rewarded by emotions during Aha! experience;

they are necessary to increase plasticity and create permanent links. I (= conscious I) have only a vague idea what goes on in my brain.

EEG and creativityEEG and creativityHow to increase cooperation between distant brain areas important for creativity?John H. Gruzelier (Imperial College), SAN President neurofeedback produced “professionally significant performance improvements” in music and dance students. Neurofeedback and heart rate variability (HRV) biofeedback. benefited performance in different ways. Musicality of violin music students was enhanced; novice singers from London music colleges after ten sessions over two months learned significantly within and between session the EEG self-regulation of ratio. The pre-post assessment involved creativity measures in improvisation, a divergent production task, and the adaptation innovation inventory. Support for associations with creativity followed improvement in creativity assessment measures of singing performance.Why? Low frequency waves = easier synchronization between distant areas; parasite oscillations decrease.

Creativity in dementia?Creativity in dementia?• Bruce L. Miller, Craig E. Hou, Emergence of Visual Creativity in

Dementia. Arch Neurol. 61, 842-844, 2004.

Miller et al (UCSF) describe a series of patients with frontotemporal dementia who acquired new artistic abilities despite evidence of deterioration in the left anterior temporal lobe. Good memory is common with frontotemporal dementia (FTD). Simple copying is typically preserved, some patients with FTD develop a new interest in painting, their artistic productivity can increase despite progression of the dementia.The artwork is approached in a compulsive manner and is often realistic or surrealistic in style.Why? Is it a disinhibition effect? Negation of linguistic concepts that block visual creativity?Slow “rewiring” of the cortex? Paradoxical functional compensation?Relation to TMS & savant syndrome studies (A. Snyder, MindLab Sydney).

Some speculationsSome speculationsHow to increase spatial coherence in the brain?

Neurofeedback, or even simpler, “mantra” meditation. Simplifies neurodynamics, stops many weaker processes that pop-up. Role of neurotransmiters in creativity? Creative people store extensive specialized knowledge in temporoparietal cortex, but may switch to divergent thinking, distant associations typical for parietal system, by modulation of the frontal lobe - locus coeruleus (norepinephrine) system. Frontal lobes are involved in working memory, divergent thinking, control of the locus coeruleus-norepinephrine system. Low levels of norepinephrine => increase synchrony, large distributed activations across brain areas, creation of novel concepts. High levels of norepinephrine (mostly from locus coeruleus), more precise memory recall, localized activations.

Ambitious Ambitious approaches…approaches…

CYC, Douglas Lenat, started in 1984. Developed by CyCorp, with 2.5 millions of assertions linking over 150.000 concepts and using thousands of micro-theories (2004).Cyc-NL is still a “potential application”, knowledge representation in frames is quite complicated and thus difficult to use.

Open Mind Common Sense Project (MIT): a WWW collaboration with over 14,000 authors, who contributed 710,000 sentences; used to generate ConceptNet, very large semantic network.Other such projects: HowNet (Chinese Academy of Science), FrameNet (Berkley), various large-scale ontologies.

The focus of these projects is to understand all relations in text/dialogue. NLP is hard and messy! Many people lost their hope that without deep embodiment we shall create good NLP systems.

Go the brain way! How does the brain do it?

Realistic goals?Realistic goals?Different applications may require different knowledge representation.Start from the simplest knowledge representation for semantic memory. Find where such representation is sufficient, understand limitations. Drawing on such semantic memory an avatar may formulate and may answer many questions that would require exponentially large number of templates in AIML or other such language.

Adding intelligence to avatars involves two major tasks:

• building semantic memory model; • provide interface for natural communication.

Goal: create 3D human head model, with speech synthesis recognition, use it to interact with Web pages local programs: a Humanized InTerface (HIT).

Control HIT actions using the knowledge from its semantic memory.

Humanized interface,search + dialogue systems

Store

Applications, eg. word games, (20Q), puzzles, creativity.

Query

Semantic memory

Parser

Part of speech tagger phrase extractor

On line dictionariesActive search and dialogues with usersManual

verification

20q for semantic data acquisition20q for semantic data acquisition

Play 20 questions with Play 20 questions with Avatar!Avatar!http://diodor.eti.pg.gda.plhttp://diodor.eti.pg.gda.pl

Think about animal – system tries Think about animal – system tries to guess it, asking no more than to guess it, asking no more than 20 questions that should be 20 questions that should be answered only with answered only with YYes or es or NNo.o. Given answers narrows the Given answers narrows the subspace of the most probable subspace of the most probable objects.objects.

System learns from the games – System learns from the games – obtains new knowledge from obtains new knowledge from interaction with the human users.interaction with the human users.

Is it vertebrate? Is it vertebrate? YYIs it mammal? Is it mammal? YYDoes it have hoof? Does it have hoof? YYIs it equine? Is it equine? NNIs it bovine? Is it bovine? NNDoes it have horn? Does it have horn? NNDoes it have long neck? Does it have long neck? YY

I guess it is I guess it is giraffegiraffe..

DREAM architecture DREAM architecture

Natural input

modules Cognitive functions

Affectivefunctions

Web/text/databases interface

Behavior control

Control of devices

Talking head

Text to speechNLP

functions

Specializedagents

DREAM is concentrated on the cognitive functions + real time control, we plan to adopt software from the HIT project for perception, NLP, and other functions.

Puzzle generatorPuzzle generatorSemantic memory may be used to invent automatically a large number of word puzzles that the avatar presents. This application selects a random concept from all concepts in the memory and searches for a minimal set of features necessary to uniquely define it; if many subsets are sufficient for unique definition one of them is selected randomly.

It has charm, it has spin, and it has charge. What is it?

It is an Amphibian, it is orange and has black spots. How do you call this animal?

A Salamander.

If you do not know, ask Google!Quark page comes at the top …

Word gamesWord gamesWord games were popular before computer games. They are essential to the development of analytical thinking. Until recently computers could not play such games.

The 20 question game may be the next great challenge for AI, because it is more realistic than the unrestricted Turing test; a World Championship with human and software players (in Singapore)?

Finding most informative questions requires knowledge and creativity.

Performance of various models of semantic memory and episodic memory may be tested in this game in a realistic, difficult application.

Asking questions to understand precisely what the user has in mind is critical for search engines and many other applications.

Creating large-scale semantic memory is a great challenge: ontologies, dictionaries (Wordnet), encyclopedias, MindNet (Microsoft), collaborative projects like Concept Net (MIT) …

Creating SMCreating SMThe API serves as a data access layer providing logical operations between raw data and higher application layers. Data stored in the database is mapped into application objects and the API allows for retrieving specific concepts/keywords.

Two major types of data sources for semantic memory:

1. machine-readable structured dictionaries directly convertible into semantic memory data structures;

2. blocks of text, definitions of concepts from dictionaries/encyclopedias.

3 machine-readable data sources are used:

• The Suggested Upper Merged Ontology (SUMO) and the the MId-Level Ontology (MILO), over 20,000 terms and 60,000 axioms.

• WordNet lexicon, more than 200,000 words-sense pairs.• ConceptNet, concise knowledgebase with 200,000 assertions.

Semantic knowledge representationSemantic knowledge representationvwCRK: certainty – truth – Concept Relation KeywordSimilar to RDF in semantic web.

Cobrais_a animalis_a beastis_a beingis_a bruteis_a creatureis_a faunais_a organismis_a reptileis_a serpentis_a snakeis_a vertebratehas bellyhas body parthas cellhas chesthas costa

Simplest rep. for massive Simplest rep. for massive evaluation/association: evaluation/association: CDV – CDV – CConcept oncept DDescription escription VVectors, forming ectors, forming Semantic MatrixSemantic Matrix

Concept Description Concept Description VectorsVectors

Drastic simplification: for some applications SM is used in a more efficient way using vector-based knowledge representation. Merging all types of relations => the most general one: “x IS RELATED TO y”, defining vector (semantic) space.

{Concept, relations} => Concept Description Vector, CDV.Binary vector, shows which properties are related or have sense for a given concept (not the same as context vector).Semantic memory => CDV matrix, very sparse, easy storage of large amounts of semantic data.

Search engines: {keywords} => concept descriptions (Web pages). CDV enable efficient implementation of reversed queries:

find a unique subsets of properties for a given concept or a class of concepts = concept higher in ontology.

What are the unique features of a sparrow? Proteoglycan? Neutrino?

Medical applications: goals & questionsMedical applications: goals & questions

• Can we capture expert’s intuition evaluating document’s similarity, finding its category? Learn form insights?

• How to include a priori knowledge in document categorization – important especially for rare disease.

• Provide unambiguous annotation of all concepts.• Acronyms/abbreviations expansion and disambiguation.• How to make inferences from the information in the text, assign values

to concepts (true, possible, unlikely, false).• How to deal with the negative knowledge (not been found, not

consistent with ...).• Automatic creation of medical billing codes from text.• Semantic search support, better specification of queries. • Question/answer system.• Integration of text analysis with molecular medicine.Provide support for billing, knowledge discovery, dialog systems.

MDS mapping of 4534 documents divided in 10 classes, using cosine distances.

1. Initial representation, 807 features.2.Enhanced by 26 selected semantic types, two steps, 2237 concepts with CC

>0.02 for at least one class.Two steps create feedback loops A B between concepts.

Structure appears ... is it interesting to experts? Are these specific subtypes (clinotypes)?

Clusterization on enhanced dataClusterization on enhanced data

Discover topics, subclusters, more focused than general categories.

Map text on the 2007 MeSH (Medical Subject Headings) ontology, more precise than ULMS.

Filter rare concepts (appearing in <1% docs) and very common concepts (>99% docs); remove documents with too few concepts (<1% of all) => smaller but better defined clusters.Leave only 26 semantic types.

Ward’s clustering used, with silhouette measure of clustering quality. Only 3 classes: two classes that mix most strongly (Pneumonia and Otitis media), add the smallest class JRA.

Initial filtering: 570 concepts with 1%<tf<99%,1002 documents.Semantic (26 types): 224 concepts, 908 docs with >1% concepts.

These 224 concepts have about 70.000 ULMS relations, only 500 belong to the 26 semantic types.

Enhancement: very restrictive, only ~25 most correlated added.

Searching for topicsSearching for topics

ResultsResultsStart, iterations 2, 3 and 4 shown, 5 clinotypes may be

distinguished.

PubMed queriesPubMed queriesSearching for: "Alzheimer disease"[MeSH Terms] AND "apolipoproteins e"[MeSH Terms] AND "humans"[MeSH Terms]Returns 2899 citations with 1924 MeSH terms. Out of 16 MeSH hierarchical trees only 4 trees have been selected: Anatomy; Diseases; Chemicals & Drugs;

Analytical, Diagnostic and Therapeutic Techniques & Equipment. The number of concepts is 1190. Loop over: Cluster analysis; Feature space enhancement through ULMS relations between MeSH concepts;Inhibition, leading to filtering of concepts.Create graphical representation.

Human categorizationHuman categorizationCategorization is quite basic, many psychological models/experiments. Multiple brain areas involved in different categorization tasks.Classical experiments on rule-based category learning: Shepard, Hovland and Jenkins (1961), replicated by Nosofsky et al. (1994).

Problems of increasing complexity; results determined by logical rules. Problems of increasing complexity; results determined by logical rules. 3 binary-valued dimensions: 3 binary-valued dimensions:

shape (square/triangle), color (black/white), size (large/small). shape (square/triangle), color (black/white), size (large/small). 4 objects in each of the two categories presented during learning. 4 objects in each of the two categories presented during learning.

Type IType I - categorization using one dimension only. - categorization using one dimension only. Type IIType II - two dim. are relevant, including exclusive or (XOR) problem. - two dim. are relevant, including exclusive or (XOR) problem. Types III, IV, and VTypes III, IV, and V - intermediate complexity between Type II - VI. - intermediate complexity between Type II - VI. All 3 dimensions relevant, "single dimension plus exception" type.All 3 dimensions relevant, "single dimension plus exception" type.Type VIType VI - most complex, 3 dimensions relevant, enumerate, no simple - most complex, 3 dimensions relevant, enumerate, no simple rule.rule.

Difficulty (number of errors made): Type I < II < III ~ IV ~ V < VIDifficulty (number of errors made): Type I < II < III ~ IV ~ V < VIFor For nn bits there are bits there are 22nn binary strings 0011…01; how complex are the rules binary strings 0011…01; how complex are the rules (logical categories) that human/animal brains still can learn? (logical categories) that human/animal brains still can learn?

Mental modelsMental models

P. Johnson-Laird, 1983 book and papers. Imagination: mental rotation, time ~ angle, about 60o/sec.Internal models of relations between objects, hypothesized to play a major role in cognition and decision-making. AI: direct representations are very useful, direct in some aspects only!

Reasoning: imaging relations, “seeing” mental picture, semantic? Systematic fallacies: a sort of cognitive illusions.

•If the test is to continue then the turbine must be rotating fast enough to generate emergency electricity.•The turbine is not rotating fast enough to generate this electricity.•What, if anything, follows? Chernobyl disaster …

If A=>B; then ~B => ~A, but only about 2/3 students answer correctly..

Kenneth Craik, 1943 book “The Nature of Kenneth Craik, 1943 book “The Nature of Explanation”, G-H Luquet attributed mental Explanation”, G-H Luquet attributed mental models to children in 1927.models to children in 1927.

Reasoning & modelsReasoning & models

Easy reasoning A=>B, B=>C, so A=>C

• All mammals suck milk.• Humans are mammals. • => Humans suck milk.

... but almost no-one can draw conclusion from:

•All academics are scientist.•No wise men is an academic.•What can we say about wise men and scientists?

Surprisingly only ~10% of students get it right, all kinds of errors!No simulations explaining why some mental models are difficult? Creativity: non-schematic thinking?

Mental models summaryMental models summary

1. MM represent explicitly what is true, but not what is false; this may lead naive reasoner into systematic error.

2. Large number of complex models => poor performance. 3. Tendency to focus on a few possible models => erroneous conclusions and irrational

decisions.

Cognitive illusions are just like visual illusions.M. Piattelli-Palmarini, Inevitable Illusions: How Mistakes of Reason Rule Our Minds

(1996)R. Pohl, Cognitive Illusions: A Handbook on Fallacies and Biases in Thinking, Judgement

and Memory (2005)

Amazing, but mental models theory ignores everything we know aboutlearning in any form! How and why do we reason the way we do? I’m innocent! My brain made me do it!

The mental model theory is an alternative to the view that The mental model theory is an alternative to the view that deduction depends on formal rules of inference.deduction depends on formal rules of inference.

Neurons learning complex logicNeurons learning complex logicBoole’an functions are difficult to learn, require combinatorial complexity; similarity is not useful, for parity all neighbors are from the wrong class. MLP networks have difficulty to learn functions that are highly non-separable.

Projection on W=(111 ... 111) gives clusters with 0, 1, 2 ... Projection on W=(111 ... 111) gives clusters with 0, 1, 2 ... nn bits; bits;solution requires abstract imagination + easy categorization.solution requires abstract imagination + easy categorization.

Ex. of 2-4D Ex. of 2-4D parity parity problems.problems.

Neural logic Neural logic can solve it can solve it without without counting; find counting; find a good point a good point of view. of view.

Parity n=9Parity n=9Simple gradient learning; quality index (abstraction of minicolumn Simple gradient learning; quality index (abstraction of minicolumn function, not single neuron) is shown below.function, not single neuron) is shown below.

Few conclusionsFew conclusionsNeurocognitive informatics: inspirations beyond perceptron.

Intuition, insight, creativity in limited domain is possible at the human competence level, opening a new vista in creativity research and suggesting new experiments.

Neurocognitive NLP leads to interesting inspirations(Sydney Lamb, Rice Univ, quite general book).

Fruitful approach: approximations to knowledge reps in brain networks: memory types/interactions, a priori knowledge, spreading activation in networks, simulations of real brain functions, graphs of consistent concepts, ontology-based enhancements, reference vectors, etc.

Drastically simplified representation of semantic knowledge is sufficient in word games, query precisiation, medical applications, common sense.

Brain processes at different levels are great inspiration! Approximate!

Thank Thank youyoufor for

lending lending your your ears ears

......

Google: W. Duch => Papers/presentations/projectsGoogle: W. Duch => Papers/presentations/projects