© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

HLC304

Building IoT Applications

with AWS and Amazon Echo

November 28, 2016

Chris McCurdy - AWS Specialist SA in Healthcare and Life Sciences

Nitin Gujral – Directory of Software Engineering and Innovation at Boston Children's

What to Expect from the Session

• What is IoT and what are some patterns that are being

used in industry?

• What are some alternatives if I just want health sensor-

related data?

• How is KidsMD using Alexa to help improve home care?

What is IoT?

What is IoT?

What is AWS IoT?

What is AWS IoT?

What is AWS IoT?

What does AWS IoT consist of?

Security First• All traffic to and from AWS IoT must be encrypted

over Transport Layer Security (TLS)• AWS Managed Certificates

• Certificate Signing Request (CSR) Certificates

• Just-in-time (JIT) Registration

• Protect your certificates• chroot

• Security Enhanced Linux (SELinux)

• One-Time Programmable (OTP) Fuses

• Trusted Platform Modules

• Smartcards

• Locks and Boxes

• FIPS-style hardware

What does AWS IoT consist of?

Thing Gateway

What does AWS IoT consist of?

Thing Gateway

Supported Protocols

HTTPS, Websockets, Secure MQTT

What is MQTT?

A lightweight pub/sub protocol, designed to minimize network bandwidth and

device resource requirements. MQTT supports TLS for encryption.

What does AWS IoT consist of?

Thing Gateway var device = awsIot.device({

keyPath: 'private_key.pem',

certPath: 'certificate.pem',

caPath: 'root.pem',

clientId: clientId,

region: 'us-east-1' });

var button = new groveSensor.GroveButton(6);

function readButtonSensorValue() {

var buttonState = button.value();

if (buttonState == 1) {

device.publish(topic, JSON.stringify({

SubmitTime: (new Date()).toJSON()}));

}

}

setInterval(readButtonSensorValue, 100);

What does AWS IoT consist of?

IoT Rules Engine

Amazon

Kinesis

Amazon S3

bucket

What does AWS IoT consist of?

IoT Rule Engine• Augment or filter data received from a device.

• Write data received from a device to an Amazon DynamoDB

database.

• Save a file to Amazon S3.

• Send a push notification to all users using Amazon SNS.

• Publish data to an Amazon SQS queue.

• Invoke a Lambda function to extract data.

• Process messages from a large number of devices using

Amazon Kinesis.

• Send data to the Amazon Elasticsearch Service.

• Capture an Amazon CloudWatch metric.

• Change an Amazon CloudWatch alarm.

• Send the data to Amazon Machine Learning to make

predictions based on an Amazon ML model.

What does AWS IoT consist of?

IoT Rule Engine

{

"sql": "SELECT * FROM 'Sensor/DispensingSensor'",

"ruleDisabled": false,

"actions": [

{

"kinesis": {

"roleArn": "arn:aws:iam::XX:role/sensor_status_role",

"streamName": "sensor_status_stream",

"partitionKey": "${ClientID}"

}

}]

}

What does AWS IoT consist of?

Thing Registry

• Associate MQTT Client IDs

• Associate Certificates

• Specify up to 3 custom attributes

What does AWS IoT consist of?

Thing Registry

{

"version": 3,

"thingName": "PortableDispenser",

"defaultClientId": "abc123",

"thingTypeName": "Dispenser",

"attributes": {

"model": "123",

"dispensingVolume": "75"

}

}

What does AWS IoT consist of?

Thing Registry

response = client.update_thing(

thingName=PortableDispenser',

attributePayload={

'attributes': {

'dispensingVolume': '150'

},

},

expectedVersion=2,

)

What does AWS IoT consist of?

Thing Shadow

• Store and retrieve current and future state of the

thing

• Simple JSON Document

• Get and set the state of a thing over MQTT or

HTTP

• Access the shadow regardless of whether the

thing is connected to the Internet

What does AWS IoT consist of?

Thing Shadow{

"state" : {

“desired" : {

"sensor" : "ON"

},

"reported" : {

"sensor" : "OFF"

},

"delta" : {

"sensor" : "ON"

} },

"version" : 10

}

What does AWS IoT consist of?

Thing Shadow

data = {

"state" : {

"desired" : {

"switch" : "on"

}

}

}

response = client.update_thing_shadow(

thingName = PortableDispenser',

payload = mypayload

)

Other services in an scalable IoT architecture

Kinesis

• Streams are made of Shards

• Each Shard ingests data up to 1 MB/sec, and up to 1000

TPS

• Each Shard emits up to 2 MB/sec

• All data is stored for 24 hours – 7 days

• Scale Kinesis streams by splitting or merging Shards

• Replay data inside of 24Hr -7days Window

Other services in an scalable IoT architecture

Amazon DynamoDB

• Provisioned at the table level• Write capacity units (WCUs) are measured in 1 KB per

second

• Read capacity units (RCUs) are measured in 4 KB per

second

• RCUs measure strictly consistent reads

• Eventually consistent reads cost ½ of constant

reads

• Read and write throughput limits are independent

• Increase as necessary, decrease at most 4 times per

UTC day

Other services in an scalable IoT architecture

Lambda

• AWS Lambda offers 23 "power levels"

• Higher levels offer more memory and more CPU

power

• 128 MB, lowest CPU power

• 1.5 GB, highest CPU power

• Compute price scales with the power level

• Duration ranging from 100 ms to 5 minutes

Example IoT architecture

IoT MQTT

protocol

IoT

certificateIoT

ruleIoT

topic

Amazon

Kinesis

AWS

Lambda Amazon

DynamoDB

Amazon

SNS

Alexa

Medication Status

monitoring device

Medication Status Backend

Node.js

AWS

LambdaAWS

Lambda

Elephant in the room

http://nos.twnsnd.co/post/104252656546/elephants-tea-party-robur-tea-room-24-march

Amazon

Kinesis

AWS

Lambda

Amazon

DynamoDB

Amazon

SNS

Alexa

AWS IoT

HIPAA Eligible Not HIPAA Eligible

Example IoT (PHI) Architecture

IoT MQTT

protocol

IoT

certificateIoT

ruleIoT

topic

Amazon

Kinesis Amazon

DynamoDB

Alexa

Medication Status

monitoring device

Medication Status Backend

Node.js

AWS

Lambda

Amazon

S3

Amazon EC2

client

Amazon EC2

AWS

Lambda

SQS

Validic Digital Health Platform

Clinical, Fitness, Wellness and

Nutritional Devices, Wearables,

Applications, and Equipment

Healthcare Providers, Clinical

Research, Wellness Apps, Insurance,

Academia

Validic architecture

3rd Party Devices,

Apps, and APIs

Validic Core Platform

VitalSnap™ OCR

Image Algorithm

Calibration

Validic

Standardization

Services

Health Data

Retrieval

Services

Admin

Database

Primary

Database

Replica

Databases

Validic API

Healthcare Systems,

Clinical Trials Platforms,

Wellness Applications

Patient / Individual

Alexa Voice Service in Healthcare

Nitin Gujral

Director of Software Engineering, Innovation and Digital Health

Boston Children’s Hospital

#1 ranked pediatric hospital

nationwide (8 out of 10

specialties)

400 plus bed; children's

hospital affiliated with Harvard

Medical School

Largest recipient of NIH funds

in pediatric-focused research

Innovation and Digital Health AcceleratorShaping the future of health care and extending Boston Children’s pediatric

leadership globally

BCH DataClinical Expertise HCIT Development

Products Platforms Ventures

Boston Children’s Expertise & Capabilities:

+Industry Partners & Startups, like:

=To launch disruptive health care focused:

Learn more:

bostonchildrens.org/accelerator

GE GrandRoundsCerner

IBM Watson Health Rock Health

Ask the Internet? –Parents are actively looking for guidance—traditional medical content sites lack

the trustworthy and readily accessible health guidance.

Although I will ALWAYS go with my gut about my kids

being sick, I typically ask the internet to confirm my

suspicion about what might be happening with them.

– Female, 36, Kids ages 1 & 4

Source: 3rd Party consumer research survey (n=200)

Decision support in consumers’ hands

Boston Children’s

Data & Expertise

Withings

Raiing

Kids.MD™ is delivering consumer facing digital health tools to patients and families across digital mediums.

Kids.MD™

powered by:

Boston Children’s Hospital

web

mobile

voice

device

research

platform

“Alexa, ask Kids.MD”

Kids.MD – Alexa Skill Architecture

AVS +

Kids.MD

“Alexa, ask Kids.MD about fever”

Skill identified in Alexa

voice service

Request sent to Kids.MD

clinical decision support

Sends

‘recommendations’ back

to Alexa voice service

Alexa converts ‘text’ to

spoken words and

streams audio

Or access

via mobile &

web app

Kids.MD™

Electronic Health

Record

Patient

Portal

Kids.MD on AWS

Kids.MD™

Amazon RDS Amazon S3

AWS Lambda API Gateway Amazon EC2

Amazon Dynamo DB

Kids.MD Skill: Development

Node Module - alexa-sdk

Handlers for intents

Node Module - alexa-sdk

Custom events

Node Module - alexa-sdk

State Machine

Context persistence using DynamoDB

Node Module - alexa-sdk

Setting context

Response

Project structure

• Structure backend code; e.g.,

handlers, model and other assets

- including tests

• Organize Alexa Skills Kit speech

assets

Debugging

Locally debug AWS Lambda code

Reference:

Blog by Nathan Grice (https://developer.amazon.com/public/community/post/Tx24Z2QZP5RRTG1/New-Alexa-

Technical-Tutorial-Debugging-AWS-Lambda-Code-Locally

• within any IDE

supporting

Node.JS

• simulate AVS

request /

response

Unit testing

Javascript testing with Jasmine (a behavior driven framework)

https://jasmine.github.io/

NPM module - jasmine or jasmine-node

Ability to mock requests and responses

Reference:

Article by Jeremy Bohrer: https://medium.com/@jjbskir/unit-testing-an-amazon-alexa-skill-with-node-js-and-

jasmine-98982544471f

Integration testing – developer tools

Lambda deployment

• NPM package

• Leverages AWS SDK

• Packaging for Lambda

• Upload to Lambda

• Supports unit testing as well

https://github.com/Tim-B/grunt-aws-lambda

Kids.MD Skill: technical

considerations

States and decision trees

Managing context in an Alexa skill

• short time frame for the duration of skill invocation

• access to user and application info

• persistent context for a user

Intent Schema Slot types

• better voice recognition

• more accuracy with user response

• supports free form responses

Custom Slot Types

Built-in Slot Types

• e.g. AMAZON.NUMBER, AMAZON.DATE,

AMAZON.DURATION

Speech Synthesis Markup Language (SSML)

• for more natural speech

• pauses and predefined formats (characters, digits, date, telephone,

address, etc.)

<s>Let us start again.</s>

<p>Would you like to hear this information again?</p>

<break strength='weak'/>

<say-as interpret-as="digits">101.4</say-as>

Speech Synthesis Markup Language (SSML)

• improved pronunciation using “phoneme”

• specifically for medical terminology

<phoneme alphabet=\"ipa\"

ph=\"sɪmp.təms\">symptoms</phoneme>

<phoneme alphabet=\"ipa\" ph=\"moətrɪŋ\">Motrin</phoneme>

Logging, auditing, and monitoring

• AWS Lambda CloudWatch logs and metrics

• Custom error logs

• Custom auditing of user interactions

• Monitoring of various service components

Account linking

• Link an Alexa user to the user of your system

• Enables contextual interactions based on user’s preferences

• Performed at the time of skill installation

• Using OAuth authorization flow

Reference:

https://developer.amazon.com/public/community/post/Tx3CX1ETRZZ2NPC/Alexa-Account-Linking-5-

Steps-to-Seamlessly-Link-Your-Alexa-Skill-with-Login-wit

Images and audio in Alexa Skills

• Display images in Cards

• JPG or PNG (less than 2 MB) Need CORS support for image URLS

• SSML Audio Tags <audio>

• MP3 support

• 90 sec limit

• Audio Streaming

• Leverage built-in intents for playback

Reference: https://developer.amazon.com/public/community/post/Tx1DSINBM8LUNHY/New-Alexa-Skills-Kit-ASK-

Feature-Audio-Streaming-in-Alexa-Skills

Voice design – lessons learned

• Maintain intuitive conversation

• provide guidance via prompts

• Balance the amount of verbal responses

• Effectively utilize Companion App(s)

• with images, audio and streaming sound

Some enterprise deployment challenges

Security

Connectivity

Voice in healthcare A new way for patients and clinicians to access health information and

personalized care in various settings

@ home

@ inpatient

@ outpatient

by patients & families

by clinicians

Context: Users:

Photo by John Blanding for The Boston Globe

Example usage:

• Home Health Hubs

• At the Bedside for Patient Care

• Virtual Assistant for Clinicians

Voice in healthcare – a new mediumStreamline and ease delivery of health care services

“Alexa, ask Children’s Hospital”

Growth of “Internet of Healthcare”

Promise of connected healthcare

Portable

Proactive

Patient Centric

Thank you!

Remember to complete

your evaluations!