new frontiers in ia: design in the era of cognitive computing

PowerPoint Presentation

New Frontiers in IADesign in the Era of Cognitive Computing

1

Imagine if you had the power to continuously gather, understand, and use all the data in the world.Imagine if you had the power to continuously gather, understand, and use all the data in the world.

Weve seen what the government would do:Prevention of terrorism, law enforcement, industrial espionage against foreign corporations, massive political eavesdropping, surveillance of Black Lives Matter.2

Could we help people work and play better?

3

Tabulation (1900-)Programmatic (1943-)Cognitive (2011-)The 3rd Era of Computing

The 3rd Era of ComputingWhat's changed is the explosion of data and the rate of change. [It] has outstripped our ability to reprogram [our] systems.

John E. Kelly IIISVP, Solutions Portfolio & Research (IBM)

Difference Engine by Charles Babbage (general purpose computing device or automatic mechanical calculator)

Hollerith, CTR, and early IBM tabulators and accounting (machines for the census).Colossus (1943, British) - first programmable (plugs and switches), electronic, digital computerEniac (1946, US) 1ST electronic general purpose-purpose computer (17,468 vacuum tubes and 5m hand-soldered parts at 30 tons, Philly lights dim4

[Cognitive computing systems] learn and interact naturally with people to extend what either humans or machine could do on their own.

Sense with networked devices

Big (global) dataData is unstructured, changes frequently, and is often conflictingSense

Learn and adapt

Model knowledge

Iterative and StatefulRemember previous interactions and use prior information Learn

Generate and evaluate hypothesesSynthesize influences, contexts, insights, ambiguous situationsThink

Natural interaction

Curious

Evolving personalities

Help define problems by asking questions and finding other inputsEngage

5

These systems are not programmed.They are systems that learn and adapt.And they interact with us in natural ways.

6

Im sorry Dave, Im afraid I cant do that.

Does anyone know who this is?Hal interacted with Dave naturally (he was not search oriented), learned and adapted, and his personality and goals evolved as new information was discovered and the mission unfolded.

We do not need to suffer the same fate as Dave at the hands of these new systems.7

8

What if technology treated you as a human being?What if technology helped you feel closer to the ones you love?What if technology helped you like a partner, rather than simply being a tool?Cynthia BreazealFounder & Chief Scientist at Jibo, Inc.

Interface

Research & Acquisitions

Project Torch

Watson - $1 billion investmentCognea- conversational AI platform (IBM: We want to create depth of personality and combine it with an understanding of the users personalities to create a new level of interaction that is far beyond todays talking smartphones.)- Close the gap to Cortana and Siri

AlchemyAPI- Provides deep learning solutions: capable of advanced data analysis such ascategorization of taxonomies, keyword extraction andsentiment analysis; close the gap to DeepMind (Watson needs programming, is only good at specific tasks)

Google NowNascent personal assistantViewdle (facial recog), Univ of Torontos DNNReseach (neural network), Wavii (language processing), Flutter (gesture recognition), Industrial Perception (computer vision)DeepMind $400m AI acquisition

Cortana- (Nov 2015) Satya Nadella predicts Cortana to replace the browser.Based on NLP, Semantic search, and Azure (cloud computing platform)

FacebookAugust FB rolled out the AI virtual assistant M inside its Messenger app.

ApplePerceptio Advanced image classifying with machine learningVocalIA AI and natural language APIEmotient AI to read emotions with facial recognition10

CaffeA modular deep learning framework (BVLC)

AppOrchid - A Silicon Startup: CC app builder for the Internet of Everything market.Enlitic deep learning healthcare company ushing in data driven medicineClarifAI AI cloud service image recognition companyLoop AI AI as a service that wants to radically change how machines autonomously learn and understand the worldGraphLab - extensible machine learning framework that enables developers and data scientists to easily build and deploy intelligent applicationsErsatz Platform for building deep neural networks in the cloud.

11

(Very) Recent NewsFacebook open sources deep learning modules from Torch AI project.(Jan 2016) Microsoft open sources deep learning toolkit available on GitHub.(Jan 2016) Yahoo open sources 13.5 TB dataset for CC research.(Dec 2015) OpenAI, a non-profit CC research effort by Musk, Thiel, Hoffman, Altman.(Dec 2015) Wikipedia uses machine learning to detect malevolent posts.(Nov 2015) Google open sources TensorFlow, an AI engine.(Nov 2015) Merrill Lynch/BoA report that the robotics/AI market to triple in 5 years.(Nov 2015) Toyota opens AI lab in Silicon Valley and Cambridge (200 researchers, $1B in 5 yrs).(Oct 2015) Intel acquires Saffron, a cognitive software maker.(Oct 2015) IBM highlights Watson ecosystem and API collection.

12

Knowledge representationProbabilisticUnstructured dataGlobal dataInteractive (engagement systems)(2011-) Cognitive Systems

Document representationDeterministicStructured dataLocal dataSearch oriented (record systems)Programmatic Systems (1943-)

Document representation with text processing with machine learning (bag-of-words, document-term matrix) to create probabilistic document descriptions.Knowledge representation using deep learning based on ontologies and taxonomies.13

Skype Translate (live voice translation system)Translation

Speech to textClosed captioningType correctionTranscription

Automatically share photos (Facebooks DeepFace)Sharing

Dynamic content (Facebook)Advertising (Facebook)Recommender Systems

Spam filteringMalevolent post detection (Wikipedias ORES)Fraud detection (AMEX*)Anomaly Detection

Use Cases & ApplicationsVoice search on Android (Google)Speech Recognition

Tagging photo collections (Baidu, Facebooks DeepFace, Google)Activity-recognition and indexing (MIT)Treezam?Face/Image Recognition

Infuse personality into virtual assistants (IBM Cognea, Siri, Cortana)Understand the personality of usersUnderstand context (Abi by Allstate)Predictive input (ClarifAI)Personal Assistance

DeepFaceInstantly shares your photos with friends and familyAutomatically show you stuff you want to see (by analyzing your daily activity)

14

Good IA Needs:People: What do they do? How do they think? What do they know? Content: What do you have? What should you have?Context: What are the goals of the solution? Who else will be involved? What are the constraints?A Practical Guide to Information Architecture by Donna Spencer (2010)

15

Re-Assessing the UserUsers have goals (forget queries)They may not know what they needFluid understanding problems and goals (as more is learned)Stymied by big data (information overload on steroids?)

Users want results and answers (forget content)Browsing knowledge, not stuff (Googles Knowledge Graph)

Context is the foreground (users are partners)Tweets, public records, blogs, purchasing patterns, friend connections, travel patterns, where we grew up, languages we speak, pets we have, education, neighborhoods we live in

Ontology becomes the core structure of a knowledge domain and appears as navigation16

New Questions: People

What are their goals? Where are they? Whos with them? Whats near them?Are they bored, scared, frustrated? Are they acting unusual? Have they tried this before?What do they really need? What kind of help are they ready for? Can I encourage them? What dont we know yet?How are they feeling?Whats the context?How far did they get? Did I help them?What do they do? How do they think? What do they know?

Re-Assessing InformationSemioticsThe study of meaning-making (signs, indication, designation, likeness, analogy, metaphor, symbolism, signification, communication).

A Mathematical Theory of Communication (Claude E. Shannon)Communication is a message conveyed with a signal.Deconstructing IA: Stepping back from linguistics

Semiotics Study of meaning-making; study of signs, indication, designation, likeness, analogy, metaphor, symbolism, signification, and communication.18

ApplicationsPresentationKnowledge ModelsOntologies, taxonomiesAnalyticsPredictive, descriptive, prescriptive, diagnosticSignal ProcessingVoice, image, networked sensorsAPIs

Generate HypothesesScoreHypotheses

Model

Elements of a cognitive system from Cognitive Computing and Big Data Analytics by Hurwitz, Kaufman, & BowlesFeature Extraction, Deep Learning, NLPAdvancing Cognitive Technologies in the EnterpriseIX & UX Opportunities

19

How could you serve your users better with cognitive technologies?

20

Thank YouPaul Michael KingLead Information ArchitectHealthwise, Inc.

21

Advancing Cognitive Technologies in the EnterpriseUX & IA Opportunities



Model

Continuously learning and adapting to us, knowledge, content, and our users.Elements of a cognitive system from Cognitive Computing and Big Data Analytics by Hurwitz, Kaufman, & BowlesFeature Extraction, Deep Learning, NLP

23

Predict search resultsType-aheadTranslationClosed captioningSpeech-to-textChat assistantsIA Contributions ApplicationsPresentationKnowledge ModelsOntologies, taxonomiesAnalyticsPredictive, descriptive, prescriptive, diagnosticSignal ProcessingVoice, image, networked sensorsAPIs


Model

Elements of a cognitive system from Cognitive Computing and Big Data Analytics by Hurwitz, Kaufman, & BowlesFeature Extraction, Deep Learning, NLP

24



Model

Elements of a cognitive system from Cognitive Computing and Big Data Analytics by Hurwitz, Kaufman, & BowlesFeature Extraction, Deep Learning, NLPAutomatic navigation

Better contextual organization

Continuous adaptation of navigation

Haptic feedbackIA Contributions

25



Model

Elements of a cognitive system from Cognitive Computing and Big Data Analytics by Hurwitz, Kaufman, & BowlesFeature Extraction, Deep Learning, NLPClassification (auto-indexing vast corpora)

Evolving classification schemas

Understanding sentiment, contextsIA Contributions

Do you have taxonomies or ontologies? An auto-categorization tool? NLP capabilities? A good analytics solution? Project managers to build out semantic and AI technologies? A forward-looking engineering team? A forgiving culture hungry for innovation?26

Being ActiveHealthy WeightHealthy EatingSleeping WellMental & Emotional HealthQuitting TobaccoSexual & Reproductive HealthDrugs & AlcoholComplementary HealthWellness

123A few hack-a-thonsOne facetA taxonomist and ontologist

It Started with a facet used for indexing content and as navigation. (Previous metadata was not suitable for navigation.)27

The 3rd Era of ComputingWhat's changed is the explosion of data and the rate of change. [It] has outstripped our ability to reprogram [our] systems.

John E. Kelly IIISVP, Solutions Portfolio & Research (IBM)Tabulation (1900-)Programmatic (1950-)Cognitive (2011-)1945ENIAC1949EDSAC1952IAS1942Atansoff Berry Computer1948Selective Sequence Electronic Calculator (IBM)1951UNIVAC1890 Census1890-1928Hollerith Tabulators1942Atansoff Berry Computer1934-1949IBM Accounting Machines1933IBM Tabulators1906 (CTR)1st Automatic Feed Tabulator1920 (CTR)1st Printing Tabulator1920Printing Tabulator (CTR)

Thermionic valvesRelaysElectromechanical Devices

28

Abbi is a chat app for helping agents in the field.Search was too difficult and agents were calling when stuck.Retrieval happens when agents get stuck. Abby understands the context and interprets ambiguous queries. (e.g., Help means more when Abby knows which field the agents cursor is stuck in.)Many agents thought Abby was human.

29

ApplicationsPresentation & VisualizationKnowledge ModelsOntologies, taxonomies, data catalogsAnalyticsPredictive, descriptive, prescriptive, diagnosticSignal ProcessingVoice, image, networked sensorsAPIsFeature ExtractionDeep LearningNLP


Model

AdaptiveLearn as information, goals, and requirements changes. Resolve ambiguity, tolerate unpredictability. Feed on dynamic data in near real time.

InteractiveInteract easily with users so they can define their needs comfortably. Interact with other devices and Cloud services.

Iterative and StatefulHelp define problems by asking questions or finding additional inputs.Remember previous interactions and use prior information.

ContextualUnderstand context (meaning, syntax, time, location, domain, regulations, user profile, task, goal). Draw on multiple information sources (structured, unstructured) and sensory inputs (visual, gestural, auditory, or sensor-provided).

30

A Practical Guide to Information Architecture by Donna Spencer (2010)People: What do they do? How do they think? What do they know? How do they feel? What are their goals? Where are they now? Do they understand the problem? Are they getting frustrated? How are they feeling? Are they bored, tense, scared? Are they acting weird? Have they tried to achieve this goal before? Were did they get stuck? Have I tried to help them before? Can I encourage them? What dont we know yet?Content: What do you have? What should you have?Can we monitor other sources for additional content? Should we think about media, sensors, prior insights? What is influencing this interaction? What does this remind me of? Context: What are the goals of the solution? Who else will be involved? What are the constraints?Do you have taxonomies or ontologies? An auto-categorization tool? NLP capabilities? A good analytics solution? Project managers to build out semantic and AI technologies? A forward-looking engineering team? A forgiving culture hungry for innovation?New Questions

31

Document representationDeterministicStructured dataLocal dataSearch oriented (record systems)Programmatic Systems (1943-)Knowledge representationProbabilisticUnstructured dataGlobal dataInteractive (engagement systems)Cognitive Systems (2011-)


The FocusEmpowering UsersMore informed decisionsBetter outcomesEmpowered EmployeesImproved performanceEasier sharing of expertiseSupport of protocolsImproved User ExperiencesNatural, personal, collaborative interactions (not search)Solutions and suggestions for achieving goals (not answers to the wrong questions)

[I]nstead of replacing human experts, cognitive systems will act as decision support systems and help users based on the best available data, whether in healthcare, finance or customer service.

Peter Fingar (The Cognitive Computing Era is Upon Us, PSFK, 20 May 2015)

Empowered Users - Enable users to find patterns in big data

Empowered Employees - Learning at the speed of light

Improved User ExperiencesText/voice apps that adapt to a users tone of voice (emotional state)Recommendations based on user personality (social media shares, purchase cycles)Engagement, not search - Infusing Cortana with personality came from one end goal: user attachment (emotions are data, emotional intelligence is computable)

Networked PersonalizationAnalysts predict that, by 2020, over 25 billion connected devices will be in use. Accommodation of owners behavior and lifestyle.

Collaborative Solution DiscoveryHypotheses and Probable SolutionsAdapt and learn from dynamic interactions33

Document representationKnowledge representationDeterministicProbabilisticStructured DataUnstructured DataLocal DataGlobal DataSearch Oriented (systems of records)Interactive (systems of engagement)Programmatic Systems (1943-)(2011-) Cognitive Systems


Re-Assessing the UserUsers have goals (forget queries)They may not know what they needFluid understanding problems and goals (as more is learned)Stymied by big data (information overload on steroids?)

Users want results and answers (forget content)Browsing knowledge, not stuff (Googles Knowledge Graph)

Context is the foreground (users are partners)Context continuously changesNeeds are ambiguous (probabilistic interpretation)How do we build models so that we show users things that help them: achieve goals, save time, or intrigue them?

Ontology becomes the core structure of a knowledge domain and appears as navigation35

The FocusMore informed decisionsBetter outcomesImproved performanceEasier sharing of expertiseSupport of protocolsNatural, personal, collaborative, comfortable interactionsSolutions and assistance (not information to queries)

[I]nstead of replacing human experts, cognitive systems will act as decision support systems and help users based on the best available data, whether in healthcare, finance or customer service.

Peter FingarThe Cognitive Computing Era is Upon Us, PSFK, May 2015

Empowered Users - Enable users to find patterns in big data

Empowered Employees - Learning at the speed of light

Improved User ExperiencesText/voice apps that adapt to a users tone of voice (emotional state)Recommendations based on user personality (social media shares, purchase cycles)Engagement, not search - Infusing Cortana with personality came from one end goal: user attachment (emotions are data, emotional intelligence is computable)

Networked PersonalizationAnalysts predict that, by 2020, over 25 billion connected devices will be in use. Accommodation of owners behavior and lifestyle.

Collaborative Solution DiscoveryHypotheses and Probable SolutionsAdapt and learn from dynamic interactions36

Document representationKnowledge representationDeterministicProbabilisticStructured DataUnstructured DataLocal DataGlobal DataSearch Oriented (systems of records)Interactive (systems of engagement)Programmatic Systems (1943-)(2011-) Cognitive Systems


null3657.1487

new frontiers in ia: design in the era of cognitive computing

Technology