Assistive Context-Aware ToolkitFelipe Pedroso

#IntelNoTDC

Agenda• Stephen Hawking and Intel• What is ACAT?• Technical Details• Architecture

• Creating Extensions• How to?• RealSense + ACAT• SharpSenses + ACAT

• Summary and Call 4 Action

#IntelNoTDC

• Professor Hawking first met Gordon Moore during a conference in 1997, where he was offered a computer with an Intel processor

• Intel upgrades his computer every couple of years and also supports him all his hardware requirements

• A cross-disciplinary team from Intel Labs developed ACAT to enable him to communicate and perform all his computer tasks more efficiently

Stephen Hawking and Intel

Images Sources: http://www.astrosurf.com/luxorion/hawking-hommage9.htm http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/

#IntelNoTDC

• Acronym for “Assistive Context-Aware Toolkit”

• Enable people with disabilities to communicate through keyboard simulation, word prediction and speech synthesis

• Developed by Intel Labs together with Professor Hawking

• Customizable and Extensible

What is ACAT?

#IntelNoTDC

Open Source

http://01.org/ACAThttps://github.com/01org/acat

#IntelNoTDC

Tech Details

• Runs on Windows 7 or higher• Code entirely written in C# using Visual 2012 (works fine on 2015)

and Microsoft .NET Framework 4.5• Microsoft Office Interop Assemblies are used to interact with Office

apps• Some tools used to develop the project:• CodeMaid• StyleCop• ReSharper

#IntelNoTDC

Architecture

Applications

ACAT Core Library

ACAT Extension Library

Extensions

Extension Extension Extension ...

#IntelNoTDC

ApplicationsACAT Qwerty ACAT Abc ACAT Tryout

Talk Window

#IntelNoTDC

ACAT Core Library - ComponentsWord

Prediction* Management

UI ManagementAnimationsPanels Widgets Themes

Text-to-Speech

ManagementTalk Window Management

*ACAT is powered by Presage, the intelligent predictive text engine created by Matteo Vescov. (http://presage.sourceforge.net)

#IntelNoTDC

ACAT Core Library - Agents

#IntelNoTDC

ACAT Core Library - Actuators

Actuator Manageme

nt

Image Sources: http://www.psfk.com/2013/02/intel-stephen-hawking.html & https://commons.wikimedia.org/wiki/File:Assistive_device.JPG

Actuators

Proximity Sensor Assistive Switch Camera

#IntelNoTDC

ACAT Extensions Library

• Allows developers to extend the functionalities from ACAT• Dynamically loaded during startup using reflection• Needs to implement specific interfaces/base classes

Extension Types

UI

Word Predictors

Agents

Actuators

Text-to-Speech Spell Checkers

#IntelNoTDC

Creating an Actuator Extension

• Create a new “Class Library” project and add references to “ACATCore.dll” and “ACATExtension.dll”

• Extend the classes “ActuatorBase” and “ActuatorSwitchBase” to interact with ACAT

• Implement the abstract methods and the use the callbacks (Init, Resume, Pause, Dispose) to run the code of your extension (threads, config, etc)

• Call the methods OnSwitchActivated (Hold/Press), OnSwitchDeactivated (Release) or OnSwitchTriggered(Click) to sinalize the interaction from the actuator

• Deploy your DLL to the extension folder and update the configuration files

#IntelNoTDC

Intel® RealSense™

Intel® RealSense™ SDK

Scene Understandin

gEnhanced

Photography

Augmented Reality

Person & Face Tracking

Gesture Recognition

Background Removal

Object Recognition and Tracking

Speech Recognition

3D Object Scanning

Hand & Blob Tracking

User Facing World Facing

#IntelNoTDC

Tracking (Position)Landmarks (78 landmarks)Pose / OrientationExpressions RecognitionPulse estimation

Intel® RealSense™ - Face Module

#IntelNoTDC

• Objective: create an actuator driven by facial expressions

• Components:• RealSenseExpressionTracker:

RealSense SDK implementation to perceive the expressions and notify the Actuator using events

• RealSenseActuator: translates the intensity of the expressions to a button behavior (pressed and released)

Intel® RealSense™ - ACAT PluginRealSense Actuator

ExtensionExpressionTracker

Actuator Switch

Actuator Manageme

nt

#IntelNoTDC

• The Camera actuator extension is a socket client that keeps looking for the “ACAT Vision” app, that has a socket server

• Simple protocol and it’s possible to customize using config files

• ACAT Vision is not yet open-source, but its code will be released in upcoming releases

Another way to connect an actuator

ACAT Vision (Server)

Camera Actuator (Client)

Actuator Manageme

nt

#IntelNoTDC

• SharpSenses* is an open source C# library that simplifies the development of Intel® RealSense™ SDK apps.

• Created and maintained by André Carlucci (Intel Black Belt & Software Innovator)

• Available on Github* and NuGet*

SharpSenses*

#IntelNoTDC

• The idea is to create a ‘fake’ server, that replicates the communication between “ACAT Vision” and ACAT

• To notify that the input was triggered, just send the string “gesture=trigger;action=CT;conf=75;time=<TIMESTAMP>”

• ACAT also sends some events back to the server (HideUI, ShowUI, ExitApp)

SharpSenses* + ACAT

ACAT Vision

Camera Actuator (Client)

Actuator Manageme

nt

‘Fake’ Server

SharpSenses

#IntelNoTDC

Summary & Call 4 Action

• ACAT is a really nice option to allow people with disabilities to communicate with others

• Open Source: You can fork and extend ACAT for your needs!• Extensions: let’s help other people with new actuators, applications

agents and new features • RealSense and SharpSenses can also drive acessibility apps.

Q/A

#IntelNoTDC

References

• Stephen Hawking and Intel• http

://iq.intel.com/behind-scenes-intel-keeps-stephen-hawking-talking/• http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/

• ACAT• https://github.com/01org/acat• http://01.org/ACAT

• RealSense: https://software.intel.com/en-us/realsense/home• SharpSenses: https://github.com/SharpSenses/SharpSenses

Thanks!

BAckup

#IntelNoTDC

ACAT Server

SharpSenses + ACAT

SharpSenses

bool canRun = true;AcatServer acatServer = new AcatServer();acatServer.Start();acatServer.AcatNotificationReceived += (sender, a) => { Console.WriteLine("Notification received from ACAT: {0}", a.NotificationType); if (a.NotificationType == AcatNotificationType.ExitApp) { canRun = false; }}; ICamera camera = Camera.Create(Capability.FaceTracking, Capability.FacialExpressionTracking);camera.Start(); camera.Face.LeftEye.Blink += (s,a) => { Console.WriteLine("SharpSenses: detected a blink from left eye."); acatServer.SendGestureMessage("CT", "trigger");}; while (canRun) { }acatServer.Stop();