Magee Campus 1 Sentiment Analysis at Ulster Dr. Paul Mc Kevitt Computer Science Research Institute School of Creative Arts & Technologies University of Ulster, Magee Derry ~ LondonDerry ~ LegenDerry Northern Ireland Magee Campus 2 Sentiment Analysis at Ulster Attitudes, emotions, feelings, moods Representation, recognition & expression of sentiment Text, audio, video, bio-signal data Emotions & mood (e.g. PlayPhysics, NewsViz) Media Asset Management (e.g. 360-MAM-Affect, 360-PlayLearn) NLP & semantics (e.g. DeepFlow) Song Form Intelligence (SoFI) Securing sentiment in video (HidInImage Ltd.) Bio-signal analysis (e.g. Galvanic Skin Response [GSR], ElectroEncephaloGram [EEG]) Magee Campus 3 Magee Campus 4 Magee Campus 5 Magee Campus 6 Magee Campus The Imagineering Quarter Physical & virtual quarter comprising 6 adjacent buildings NWRC (3), Magee (3), The Nerve Centre, FAB-LAB, Verbal Arts Centre, LYIT Digital Creativity (games, film/TV, music/radio, drama/dance, art/design, virtual worlds, digital storytelling) 7 Magee Campus 8 Magee Campus 9 Magee Campus 10 Magee Campus 11 Magee Campus Imagineering Quarter corridors Access to Project Kelvin (secure, high capacity [10 G. LanPhy], delay [2 ms.]) Digital Derry & CultureTECH Festival Tech City UK (twin-city London), MediaCityUK (Salford), Scotland Creative Clyde (Glasgow) & Digital Hub Ireland (Dublin) MIT Media Lab & USC Institute for Creative Technologies (USA), ITESM-CCM, Monterrey (Mexico) Aalborg University, CADIA Reykjavik University Karolinska Institute Stockholm & Media Lab Helsinki (Scandinavia) LIMSI-CNRS (France), Bremen, Bielefeld & Freiberg Universities (Germany), Granada University (Spain) Universiti Kebangsaan (Malaysia), National Institute of Design (India), Nihon University (Japan) & The Confucius Institute (China) 12 Magee Campus International Loebner Prize 2013 Contest in Artificial Intelligence (AI) Saturday, 14th September, 2013 Derry ~ LondonDerry ~ LegenDerry CultureTECH 2013 (12-15th September, 2013) The Turing Test $6000 in prizes sponsored by New York Philanthropist, Dr. Hugh Loebner WGBH (Boston, USA) & 360 Production (Derry/London) documentary on Artificial Intelligence (AI) for Nova Primetime Science TV Series on Public Broadcasting Service (PBS) 13 Magee Campus 14 Magee Campus 15 CHAMELEON: a platform for developing intelligent multimedia applications Speech input/output, image processing & laser IntelliMedia Workbench campus information system application Semantic representation with frames (Minsky, 1975) Blackboard for semantic representation Bielefeld DACS distributed communications platform (Fink et al., 1996) Implemented with C, C++, DACS, Infovox TTS, grapHvite Applications in multimodal systems development Magee Campus Architecture of CHAMELEON 16 Magee Campus IntelliMedia Workbench 17 Magee Campus CHAMELEON Semantic Representation Semantic Frames (Minsky, 1975) 18 Input frame: [SPEECH-RECOGNISER UTTERANCE: (Point to Hanne's office) INTENTION: instruction! TIME:timestamp] Output frame: [LASER INTENTION: description (pointing) LOCATION: coordinates (5, 2) TIME: timestamp] Integration frame: [NLP INTENTION: description (pointing) LOCATION: office (tenant Hanne) (coordinates (5, 2)) UTTERANCE: (This is Hanne's office) TIME: timestamp] Magee Campus Computational model of emotion Implements Control-Value Theory of Emotions (Pekrun et al. 2007) Representing, recognising & inferring emotions PlayPhysics game-based virtual learning environment Results promising with Cohen's Kappa in range Magee Campus Emotion representation & recognition Emotion representation with Dynamic Bayesian Networks (DBNs) DBN skeletons derived employing Probabilistic Relational Models (PRMs) DBNs completed using Necessary Path Condition (NPC) algorithm with Pearson Correlations & Binary/Multinomial Logistic Regression (BLR, MLR) Expectation Maximisation (EM) algorithm employed for DBN parametric learning Emotion recognition with Contextual Variables (e.g. help request, mouse focus, timing) & bio-signal Galvanic Skin Response (GSR) 20 Magee Campus Olympia architecture 21 Magee Campus PlayPhysics system Focus on 1st year University Physics (Kinematics and Dynamics, Newton's Laws [particles, rigid bodies], 3D Vector Mechanics) PlayPhysics implemented with Unity 3D, Hugin Lite, Java, HTML, 3D studio Max, Poser, LightWave, Spacescape, JavaScript, Cascading Style Sheets (CSS) & XML Bluetooth GSR sensor using LEGO NXT intelligent brick & 8528 converter cable M8 robot learning companion implemented with synchronised animations and pre-defined text M8 robot provides instruction, prompting emotion reporting, motivating, praising & conveying affinity 22 Magee Campus 23 Magee Campus Visualising emotions & mood Recognising emotion & mood in online news articles 2D visualisation of emotion & mood Natural Language Processing (NLP) NewsViz visualisation of online soccer reports Promising results comparable to human analysis with detection of bias in soccer reports 24 Magee Campus Recognising emotions & mood Word lexicon with emotion indices Detection of emotion in Parts-of-Speech (POS) (adjectives, nouns) Adjectives, nouns labelled with emotion & intensity value Recurring emotions determine moods (minimum threshold) Use of different methods (word-by-word, sentence, minimum threshold) NLP option selection (POS, words/sentences, domain/linguistic knowledge, emotion degree) 25 Magee Campus News Visualisation Emotion Extraction NewsViz architecture Magee Campus News Visualisation Emotion Extraction Emotion visualiser Magee Campus NewsViz system NLP POS Tagging Adjective/Noun detector applied to POS text Emotion word selector employing WordNet with emotion indices Emotion summariser employing football & linguistic knowledge Animation simulator employing graphics database 28 Magee Campus Graphics Database Abstract Visuals for 4 Emotions 2 - boring 4 - happy 3 - tense 1 - sad News Visualisation Emotion Extraction Magee Campus 30 Magee Campus Media Asset Management (MAM) MAM content selected by sentiment analysis Sentiment analysis applied to media analytics Machine Learning (ML) applied to viewer text comments YouTube EDU video content 360-MAM-Affect sentiment analysis & prediction 360-PlayLearn online virtual learning environment Results promising and comparable to humans with initial training corpus of 200 comments 31 Magee Campus 360-PlayLearn 360-PlayLearn online virtual learning environment Employment of gamification & game-based learning Underpinned by 360-MAM & 360-Cloud 360-MAM employing Razuna 360-Cloud employing Google App Engine Deployment on desktop, mobile & TV devices 32 Magee Campus 360-PlayLearn architecture 33 Magee Campus 360-MAM-Affect architecture 34 Magee Campus 360-MAM-Affect system 360-MAM-Affect Prediction Module Deployment of Google Prediction API Text positivity rating (+ve [5], -ve [-5], neutral [0]) Viewer comments ML training corpus from YouTube EDU Data API Comments stored in comment library with App Engine Datastore 35 Magee Campus 36 Magee Campus NLP & semantics Argumentation extraction & reasoning Requirements specifications, design documents, marketing material,s, Blogs DeepFlow design decision support for manufacturing Document representation with similarity based indexing Employment of GATE NLP tool & POS Tagging/Parsing 37 Magee Campus Magee Campus 39 Magee Campus SoFI: Song Form Intelligence for repairing streaming music across wireless bursty networks Large packet loss dropouts of s. Use audio pattern recognition & matching to repair gaps K-means clustering classifier determines self-similarity matches Match current sections of song received with previous sections played Typicality of Western Tonal Format (WTF) Use of MPEG-7 Audio Spectrum Envelope (ASE) multimedia content description standard giving features for extraction Implemented with G-streamer for steaming media, Ices2, Icecast2 for audio broadcast & MPEG-7 Audio Spectrum Envelope (ASE) Applications in repair of streaming music 40 Magee Campus Architecture of SoFI 41 Magee Campus Spectrogram representation 42 Magee Campus 43 Magee Campus 44 Magee Campus Securing sentiment in video Securing digital media content Encryption & hiding of information (text, images) inside images employing steganography Unique software algorithm for encryption Novel technique for embedding data in images & video Robust to image/data compression Integration within other document management solutions such as a `secure layer HidInImage Ltd. spin-out company 45 Magee Campus 46 Steganography Key Encoder Cover Image Secret Data Stego Object Decoder Secret Data Communications Channel Magee Campus 47 Embedding EPRs data in innocuous looking image: (a) shows a CT scan image of a patient with her information details, (b) encrypted secret data (payload) of (a), a clean image showing nature in which the encrypted data will be embedded to and finally (d) shows the stego-image carrying the encrypted patient data Secret Communication: Example Magee Campus 48 Secret Data Cover Image Stego Object Document authentication Magee Campus 49 How to detect alterations Altered Document Recovered Document Grayscale image of Altered Document Final Result Magee Campus Bio-signal analysis Bio-signal analysis of emotional response Galvanic Skin Response [GSR] Bio-signal analysis (e.g. PlayPhysics) Electroencephalogram (EEG) Bio-signal analysis (e.g. BCI SpaceGame) EEG-based non-invasive BCI 50 Magee Campus Galvanic Skin Response (GSR) Emotion recognition in PlayPhysics Bluetooth GSR sensor using LEGO NXT intelligent brick Foil paper, loop & hoop & 8528 converter cable Belkin Mini-Bluetooth adapter (USB 2.0) Sensor connected to index and medium fingers of least-favoured hand 51 Magee Campus 52 Magee Campus ElectroEncephaloGram (EEG) Non-invasive brain-computer interface (BCI) in SpaceGame Does not rely on neuromuscular control Alternative communicatory and control mechanisms Three channels of EEG are recorded from the scalp User imagines left and right hand movement (motor imagery) 4 game levels (easy --> impossible) 53 Magee Campus BCI Subject-specific frequency bands selected using PSO Pre-processing : NTSPP-SF-CSP (Coyle el al, 2005, 2008, 2009 ) Features: log-variance of surrogate signals (2s sliding window) Classifier : LDA (D>0 Hand or D