computing at scale

Computing at Scale: Data Exploration Jerjou Cheng, Barry Brumitt

Upload: jerjou

Post on 27-Jan-2015

114 views

Category:

Technology

3 download

Report

Download

Tags:

Embed Size (px):

DESCRIPTION

Talk given at Seattle Tech Forum on Dev 15, 2010 at Bellevue City Hall.

TRANSCRIPT

Computing at Scale:Data Exploration

Jerjou Cheng, Barry Brumitt

HELLO

/jur'jō/ɯoɔ˙ǝlƃooƃ@noɾɹǝɾ

MY NAME IS

Developer Programs Engineer(Developer Relations)

Google Storage

Michael PIERROThttp://www.freephotobank.org/v/sky-stars/cloud/Cloud-19.jpg.html

Computing at Scale:Data Exploration

Jerjou Cheng, Barry Brumitt

Overview

• Google App Engine• Google Storage for Developers• Prediction API• BigQuery

Introductions

Who are theseservices for?

A World without Clouds

• Build a web applicationoStartup costsoMaintenance / reliabilityoScaling

Michael Scheltgenflickr.com/mscheltgen/

Google App Engine

Easy to startEasy to maintainEasy to scale

Users

gigy Socialize - traffic

Overview

• Google App Engine• Google Storage for Developers• Prediction API• BigQuery

Overview

• Google App Engine• Google Storage for Developers• Prediction API• BigQuery

A World without Clouds

• Store data• Reliability• Sharing• Large objects

Michael Scheltgenflickr.com/mscheltgen/

Google Storage for Developers

• Google infrastructure• You control access to your data• Store massive data in Google's cloud• Easy interface

Example

Internal use cases

• Content hosting

• Sharing

• Data Import Google BigQuery

Google Prediction API

Overview

• Google App Engine• Google Storage for Developers• Prediction API• BigQuery

Prediction API

• Cloud-hosted machine learning as service• Simple interface over complex analysis• Predict results in real-time

How does it work?

"english" The quick brown fox jumped over the lazy dog.

"english" To err is human, but to really foul things up you need a computer.

"spanish" No hay mal que por bien no venga.

"spanish" La tercera es la vencida.

"english" To be or not to be, that is the question.

"spanish" La fe mueve montañas.

The Prediction APIfinds relevantfeatures in the sample data during training.

The Prediction APIlater searches forthose featuresduring prediction.

Prediction API

1. Upload

2. Train

Upload your training data toGoogle Storage

Build a model from your data

Make new predictions

prediction/v1.1/training?data={}POST : a training request

prediction/v1.1/training/{}/predictGET : model infoPOST : a prediction request

Use the API, gsutil or any compatible utility to upload your data to Google Storage

3. Predict

Example

Prediction API

• Google's machine learning algorithms• Available as RESTful HTTP service• Predict results in real-time

Overview

• Google App Engine• Google Storage for Developers• Prediction API• BigQuery

GET /information HTTP/1.0

• To request access and get more information, go to:o http://code.google.com/appengineo http://code.google.com/apis/bigqueryo http://code.google.com/apis/predicto http://code.google.com/apis/storage

2015-05-20 At Scale Enterprise Computing v2.ppt

Computing Recommendations at Extreme Scale with Apache Flink

HC3 - Scale Computing

Distributed Computing Data Profiling at Scale Dr. Thorsten ... · Distributed Computing Data Profiling at Scale Dr. Thorsten Papenbrock Information Systems Group, HPI . Dr. Thorsten

Computing at Massive Scale: Scalability and …act.buaa.edu.cn/yangrenyu/paper/sose2016.pdfComputing at Massive Scale: Scalability and Dependability Challenges Renyu Yang† and Jie

CSci5221: Data Centers, Cloud Computing, … Cloud Computing and Data Centers: Overview What’s Cloud Computing? Data Centers and “Computing at Scale” Case

NSF Workshop on Pervasive Computing at Scale, PeCSsensorlab.cs.dartmouth.edu/NSFPervasiveComputingAtScale/pdf/final... · NSF Workshop on Pervasive Computing at Scale, PeCS The NSF

Computing for the Future of the Planet (at Scale) Computing for the Future of the Planet (at Scale) Andy Hopper The Computer Laboratory University of Cambridge

WINTER Template Distributed Computing at Web Scale Kyonggi University. DBLAB. Haesung Lee

SECURITY AT SCALE WITH CLOUD COMPUTING · Security at Scale with Cloud Computing 5 Our open-source version of these containers, Kubernetes, is a popular choice for developing and

Large-Scale Computing with Grids

Accelerated Computing on AWS - ASAP 2018asapconference.org/slides/amazon.pdf · Accelerated Computing on AWS Applications for GPUs and FPGAs F1. Data Drives Computing at Scale Why

Scale computing - Unique Networks

Visualizing Science at Scale with Longhorn - Kelly Gaither, Texas Advanced Computing Center

Google Cloud computing at scale. o Punch scaled test … · 2020. 3. 9. · Google Cloud computing at scale. Google Cloud Platform engineering, Platform engineering, Developer Operations,

Profiling a warehouse-scale computerskanev/papers/isca15wsc.pdfProfiling a warehouse-scale computer ... sands of computing nodes for performance and cost-efficiency at scale. Patterson

SECURITY AT SCALE WITH CLOUD COMPUTING · 2019. 2. 26. · Security at Scale with Cloud Computing 3 CONSIDER WHAT GOOGLE DOES EVERY MINUTE OF THE DAY 10 MILLION 400 MILLION+ 694,000

Computing at Massive Scale: Scalability and Dependability ... · Computing at Massive Scale: Scalability and Dependability Challenges Renyu Yang† and Jie Xu *† †School of Computing,

Computing for the Future of the Planet (at Scale) Andy Hopper

DOE Extreme Scale Computing

Powering Applications at Every Scale: The Next Wave of Unified Computing Innovation

Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015

Cloud Computing Economies of Scale

Extreme-scale computing systems – High performance computing systems Current No. 1 supercomputer Tianhe-2 at 33.86 petaflops Pushing toward exa-scale computing

Petabyte-scale computing for LHC

CS5950 / CS6030 Cloud Computing - wmich.edu · • AWS starter – EC2, VPC, SecurityGroups, Storage • Computing at scale – The need for scalability; scale of current services

mapping brain activity at scale with cluster computing...mapping brain activity at scale with cluster computing Jeremy Freeman 1, Nikita Vladimirov 1, Takashi Kawashima , Yu Mu , Nicholas

A small-scale testbed for large-scale reliable computing

SECURITY AT SCALE WITH CLOUD COMPUTING - Google Cloud · Security at Scale with Cloud Computing 2 ... Understanding how we’ve built and evolved our defenses in ... That keeps our

NSF Workshop on Pervasive Computing at Scale, PeCScampbell/NSF-PeCS-finalreport.pdf · NSF Workshop on Pervasive Computing at Scale, PeCS The NSF sponsored workshop on Pervasive Computing

Computing Constrained Shortest-Paths at Scale · 2018. 3. 14. · Computing Constrained Shortest-Paths at Scale ALBERTO VERA, Cornell University SIDDHARTHA BANERJEE, Cornell University

2015-05-20 At Scale Enterprise Computing v2.pptcloser.scitevents.org/Documents/Previous_Invited_Speakers/2015/... · At Scale Enterprise Computing Dr. Chung-Sheng Li ... ESB, BPM,

IT@Intel: Green Computing at Scale White Paper

Interactively Visualizing Science at Scale · Interactively Visualizing Science at Scale Kelly Gaither Director of Visualization/Senior Research Scientist . Texas Advanced Computing

Exa-Scale Volunteer Computing

computing at scale

Technology

prediction request

training data

prediction api later

data store massive data

clouds store data reliability

data exploration jerjou

comappengine http

comapisbigquery http