machine learning with google machine learning apis - puppy or muffin?
TRANSCRIPT
a.k.a. “Puppy or muffin?”
Machine learning with Google APIsEasily leverage machine learning in your apps
Bret McGowen@bretmcg
2@bretmcg
Who am I?
Developer Advocate, Google Cloud PlatformBret McGowen / @bretmcg
● New York, NY● Aspiring Node.js developer● College football fan (Gig 'em, Aggies!)● Lord of the Rings aficionado
What we’ll cover
01
02
03
04
05
A (very) brief overview of machine learning
Machine learning at Google and elsewhere
Vision API
Speech API
Natural Language API
01 A (very) brief overview of machine learning
5@bretmcg 5
Machine learning is learning from examples and experience.
6@bretmcg
Machine learningArtificial intelligenceMaking machines intelligent.
Making machines that learn.
"Programming a computer to learn to be clever is easier than programming a computer to be clever directly."
7@bretmcg
Google Cloud Platform 8
Why the sudden explosion in machine learning?
9@bretmcg
The bigger, the better
From: Andrew Ng
12@bretmcg
13@bretmcg
DNN requires large training datasets
Large models don't fit into a GPU
Requires try-and-errors to find the best design, configs and params
↓
Need to spend a few days or weeks to finish a training
The challenge: computing power
GPUs run at nanosecondsGPU cluster needs microsec network
02 Machine Learning at Google (and elsewhere)
17@bretmcgEnterprise
Google Cloud is
The Datacenter as a Computer
18@bretmcg
Jupiter Network
• 10 GbE x 100 K = 1 Pbps
• Consolidates servers with
microsec latency
19@bretmcg
Tensor Processing UnitASIC for TensorFlow
Designed by Google
10x better perf / watt
latency and efficiency
bit quantization
20@bretmcg
TPUs in ProductionRankBrain
AlphaGo
Google Photos
Speech
and more
21@bretmcg
22@bretmcg
signalfor Search ranking,
out of hundreds
improvementto ranking quality
in 2+ years
#3 #1
Search
machine learning for search engines
RankBrain: a deep neural network for search ranking
WaveNet by
Google DeepMind
Combined vision and translation
Smart reply in Inbox by Gmail
10%of all responses sent on mobile
30@bretmcg
Google Photos
Open source Machine Learning library
Especially useful forDeep Learning
For research and production
Apache 2.0 license
TensorFlow
powered
Fried Chicken
Nugget Server
From: http://www.rt-net.jp/karaage1/
TensorFlow powered Cucumber Sorter
TensorFlow＋
RasPi forsorting garbages
From: https://techcrunch.com/2016/09/13/auto-trash-sorts-garbage-automatically-at-the-techcrunch-disrupt-hackathon/
Google Cloud Platform 36
Let's try some human-powered image detection
Puppy
or
muffin?
source: boredpanda.com
source: boredpanda.com
Google Cloud Platform 40
What if I'm not a machine learning expert?
41@bretmcg
The Machine Learning Spectrum
TensorFlow Cloud Machine Learning Machine Learning APIs
BYOML skills (Developers, Data Scientists)
(Friendly Machine Lea
rning)
Pre-packaged ML
03 The Cloud Vision API Complex image detection with a simple REST request
Google Cloud Platform 44
Making an API request
Making a request{ "requests":[ { "image": { "content": "base64ImageString"
// Alternatively, you can pass a Google Cloud Storage url here }, "features": [ { "type": "LABEL_DETECTION", "maxResults": 10 }, { "type": "FACE_DETECTION", "maxResults": 10 },
// More feature detection types... ] } ]}
Google Cloud Platform 46
Let’s see some JSON responses
47
{ "labelAnnotations" : [ { "mid" : "\/m\/01wydv", "score" : 0.92442685, "description" : "beignet" }, { "mid" : "\/m\/0270h", "score" : 0.90845567, "description" : "dessert" }, { "mid" : "\/m\/033nb2", "score" : 0.74553984, "description" : "profiterole" }, { "mid" : "\/m\/01dk8s", "score" : 0.71415579, "description" : "powdered sugar" } ] }
Label Detection
47
"landmarkAnnotations": [
{
"mid": "/m/0c7ln",
"description": "Navy Pier",
"score": 36,
"boundingPoly": {
"vertices": [
{
"x": 275,
"y": 102
}, //...
]
},
"locations": [
{
"latLng": {
"latitude": 41.888685,
"longitude": -87.601311
}
}
], //...
},
{
"mid": "/m/01_d4",
"description": "Chicago",
"score": 31,
"boundingPoly": {
"vertices": [
{
"x": 1086,
"y": 346
},
{
"x": 1496,
"y": 346
},
{
"x": 1496,
"y": 832
},
{
"x": 1086,
"y": 832
}
]
},
"locations": [
{
"latLng": {
"latitude": 41.866724,
"longitude": -87.60852
}
}
],
"lat": 41.866724,
"long": -87.60852,
"latText": "41.866724",
"longText": "-87.608520"
},
{
"mid": "/m/06_7k",
"description": "Chicago",
"score": 23,
"boundingPoly": {
"vertices": [
{
"x": 1194,
"y": 344
},
{
"x": 1537,
"y": 344
},
{
"x": 1537,
"y": 899
},
{
"x": 1194,
"y": 899
}
]
},
"locations": [
{
"latLng": {
"latitude": 41.889232,
"longitude": -87.62312299999999
}
}
],
"lat": 41.889232,
"long": -87.62312299999999,
"latText": "41.889232",
"longText": "-87.623123"
}
],
Landmark Detection
48
49
..."itemListElement": [ { "@type": "EntitySearchResult", "result": { "@id": "kg:/m/0c7ln", "name": "Navy Pier", "@type": [ "Thing", "Place", "LandmarksOrHistoricalBuildings", "TouristAttraction" ], ...
"detailedDescription": { "articleBody": "Navy Pier is a 3,300-foot-long pier on the Chicago shoreline of Lake Michigan. It is located in the Streeterville neighborhood of the Near North Side community area.", "url": "http://en.wikipedia.org/wiki/Navy_Pier"
...
Knowledge Graph sidebarGET https://kgsearch.googleapis.com/v1/entities:search?ids=%2Fm%2F0b__kbm&key={API_KEY}
50
"textAnnotations": [ { "locale": "en", "description": "U.S. COAST GUARD AUXILIARY\n242039\n", "boundingPoly": { "vertices": [ { "x": 429, "y": 307 }, { "x": 1178, "y": 307 }, { "x": 1178, "y": 770 }, { "x": 429, "y": 770 } ] } }, // ... ]
Text Detection
51
"faceAnnotations" : [
{
"headwearLikelihood" : "VERY_LIKELY",
"surpriseLikelihood" : "VERY_UNLIKELY",
"rollAngle" : 2.8030474,
"angerLikelihood" : "VERY_UNLIKELY",
"landmarks" : [
{
"type" : "LEFT_EYE",
"position" : {
"x" : 221.60617,
"y" : 638.263,
"z" : 0.0017568493
}
},
...
],
"boundingPoly" : {
"vertices" : [
{
"x" : 89,
"y" : 436
},
...
Face Detection
"detectionConfidence" : 0.98838496,
"joyLikelihood" : "VERY_LIKELY",
"panAngle" : -1.0822374,
"sorrowLikelihood" : "VERY_UNLIKELY",
"tiltAngle" : -2.5003448,
"underExposedLikelihood" : "VERY_UNLIKELY",
"blurredLikelihood" : "VERY_UNLIKELY"
52
"faceAnnotations" : [
{
"headwearLikelihood" : "VERY_UNLIKELY",
"surpriseLikelihood" : "VERY_UNLIKELY",
rollAngle" : -4.6490049,
"angerLikelihood" : "VERY_UNLIKELY",
"landmarks" : [
{
"type" : "LEFT_EYE",
"position" : {
"x" : 691.97974,
"y" : 373.11096,
"z" : 0.000037421443
}
},
...
],
"boundingPoly" : {
"vertices" : [
{
"x" : 743,
"y" : 449
},
...
Face Detection
"detectionConfidence" : 0.93568963,
"joyLikelihood" : "VERY_LIKELY",
"panAngle" : 4.150538,
"sorrowLikelihood" : "VERY_UNLIKELY",
"tiltAngle" : -19.377356,
"underExposedLikelihood" : "VERY_UNLIKELY",
"blurredLikelihood" : "VERY_UNLIKELY"
53
"labelAnnotations" : [
{
"mid" : "\/m\/01fklc",
"score" : 0.9337945,
"description" : "pink"
},
{
"mid" : "\/m\/09g5pq",
"score" : 0.83878618,
"description" : "people"
},
{
"mid" : "\/m\/017ftj",
"score" : 0.71847415,
"description" : "sunglasses"
},
{
"mid" : "\/m\/019nj4",
"score" : 0.69381392,
"description" : "smile"
}
]
Putting it all together: face + label + landmark "landmarkAnnotations" : [
{
"boundingPoly" : {
"vertices" : [
{
"x" : 153,
"y" : 64
},
...
]
},
"mid" : "\/m\/0c7zy",
"score" : 0.56636304,
"description" : "Petra",
"locations" : [
{
"latLng" : {
"longitude" : 35.449361,
"latitude" : 30.323975
}
}
]
}
]
54
"safeSearchAnnotation" : {
"spoof" : "VERY_UNLIKELY",
"medical" : "VERY_UNLIKELY",
"adult" : "VERY_UNLIKELY",
"violence" : "VERY_UNLIKELY"
}
But wait...is it appropriate?
Live Demo
04 The Speech API Speech to text transcription in over 80 languages
57@bretmcg
What can I do with the Speech API?● Speech to text transcription in over 80 languages
● Supports streaming and non-streaming recognition
● Filters inappropriate content
58@bretmcg
Translation Response "responses": [{
"results": [{
"alternatives": [
{
"transcript": "how old is the Brooklyn Bridge",
"confidence": 0.987629
}],
"isFinal": true
}
]
}]
59@bretmcg
Let’s make a recording!
1. Make a recording using SoX, a command line utility
for audio files
2. Base64 encode the recording
3. Build our API request in a JSON file
4. Send the JSON request to the Speech API
Bash script at https://goo.gl/zgvbWY
05 Cloud Natural Language API Perform sentiment analysis and entity recognition on text
61@bretmcg
What can I do with the Natural Language API?Three methods:
1. Analyze entities - The Washington Nationals are an MLB team
from Washington, D.C.
2. Analyze sentiment - The DC metro area is a great place to live
and work.
3. Analyze syntax - Michelle Obama is married to Barack Obama
62@bretmcg
Analyze Entities
There is plenty to do in Washington, D.C. without having to
spend a single dollar. The Smithsonian is totally free, as is
the U.S. Capitol building, along with the White House
tour, the U.S. Supreme Court Building, and many other
monuments. Bring your Android camera and take photos!
-- Bret McGowen's Made-Up Guide to Washington, D.C.
63@bretmcg
Analyze Entities
There is plenty to do in Washington, D.C. without having to
spend a single dollar. The Smithsonian is totally free, as is
the U.S. Capitol building, along with the White House
tour, the U.S. Supreme Court Building, and many other
monuments. Bring your Android camera and take photos!
-- Bret McGowen's Made-Up Guide to Washington, D.C.
64@bretmcg
Analyze Entities
"name": "Bret McGowen","type": "PERSON","metadata": {},"salience": 0.0164788,"mentions": [ { "text": { "content": "Bret McGowen", "beginOffset": -1 } }]
There is plenty to do in Washington, D.C. without having to
spend a single dollar. The Smithsonian is totally free, as is
the U.S. Capitol building, along with the White House
tour, the U.S. Supreme Court Building, and many other
monuments. Bring your Android camera and take photos!
-- Bret McGowen's Made-Up Guide to Washington, D.C.
"name": "Washington, D.C.","type": "LOCATION","metadata": { "wikipedia_url": "http://en.wikipedia.org/wiki/Washington,_D.C."},"salience": 0.29157177, "mentions": [ { "text": { "content": "Washington, D.C.", "beginOffset": -1 } }
"name": "Android","type": "CONSUMER_GOOD","metadata": { "wikipedia_url": "http://en.wikipedia.org/wiki/Android_(operating_system)" }
65@bretmcg
Analyze Sentiment
Texas is the best state in the Union.
{
“documentSentiment”: {
“polarity”: 1,
“magnitude”: 0.8
}
}
66@bretmcg
Analyze Syntax
“My dog Fluffy has been missing ever since Bret brought in that
basket of muffins.”
67@bretmcg
Analyze Syntax
68@bretmcg
Let’s see a demo!
Twitter Streaming NL API BigQuery
bit.ly/nl-olympics
69@bretmcg
APIs we covered● Vision: cloud.google.com/vision
● Speech: cloud.google.com/speech
● Natural Language: cloud.google.com/natural-language
Related APIs:
● Translate: cloud.google.com/translate
● Prediction: cloud.google.com/prediction
● Knowledge Graph API: developers.google.com/knowledge-graph
70@bretmcg 70
Oh, in case you were wondering...
source: boredpanda.com
"labelAnnotations": [
{
"mid": "/m/02wbm",
"description": "Food",
"score": 96
},
{
"mid": "/m/02q08p0",
"description": "Dish",
"score": 87
},
{
"mid": "/m/0270h",
"description": "Dessert",
"score": 86
},
{
"mid": "/m/0dxb5",
"description": "Berry",
"score": 83
},
{
"mid": "/m/052lwg6",
"description": "Baked Goods",
"score": 78
},
{
"mid": "/m/0hz4q",
"description": "Breakfast",
"score": 76
},
{
"mid": "/m/01tcjp",
"description": "Muffin",
"score": 76
},
source: boredpanda.com
source: boredpanda.com
?"Baked goods"
"Mammal"
Looking for:
● Dog (or breed)
● Muffin
74@bretmcg 74
Bret McGowen@bretmcg
Thank You!