cloudera world tokyo 2014 ltセッション「マイクロソフトとhadoop」

Download Cloudera World Tokyo 2014 LTセッション「マイクロソフトとHadoop」

Post on 09-Jul-2015

371 views

Category:

Technology

0 download

Embed Size (px)

DESCRIPTION

Cloudera World Tokyo 2014のライトニングトークセッション資料です。

TRANSCRIPT

  • Hadoop

    2014116 (@ksasakims)

  • 2(@ksasakims)

  • 3

  • 4Hadoop

  • 5

  • Hadoop

    2014116 (@ksasakims)

  • 8Dryad

    http://research.microsoft.com/en-us/projects/dryad

  • 9Cosmos

    http://research.microsoft.com/en-us/events/fs2011/helland_cosmos_big_data_and_big_challenges.pdf

    PB

    ()

    Dryad

    SCOPE SQL Dryad

  • 10

    Cosmos

    Dryad

    Hive, Pig

    HDFS

  • 11

    SCOPE

    http://research.microsoft.com/en-us/um/people/jrzhou/pub/Scope.pdf

  • 12

    AdCenter, Azure, Bing, Bing Maps, MSN, Xbox Live

    and more!

    2005

    Cosmos

  • 13

  • 14http://www.publickey1.jp/blog/10/hadoopdryad.html

    http://blogs.technet.com/b/windowshpc/archive/2011/05/20/dryad-becomes-linq-to-hpc.aspx

  • 15

    LINQ to HPC

    2

    LINQ to HPCWindows Hadoop

  • 17

    Microsoft Azure HDInsight Service

    Hadoop

    2013 10 11

    OS

    Windows Server 2012 R2

    3 HDInsight-3.0

    (HDP 2.0 / Hadoop 2.2)

    HDInsight-3.1

    (HDP 2.1 / Hadoop 2.4)

  • 18

    HDInsight Service

    BLOB

    A48 /

    A34 /

    Microsoft Azure PaaS BLOB

    Worker

    Microsoft Azure BLOB HDFS

  • 19

    Azure

    http://sigops.org/sosp/sosp11/current/2011-Cascais/printable/11-calder.pdf

  • 20

    (BLOB)

  • 21

    NativeAzureFileSystem

    BLOB Hadoop

    Hadoop

    NativeAzureFileSystem Azure BLOB

    URI

    Java(org.apache.Hadoop)

    Local file fs.LocalFileSystem

    HDFS hdfs hdfs.DistributedFileSystem HDFS

    FTP ftp fs.ftp.FTPFileSystem FTP

    S3 s3n fs.s3native.NativeS3FileSystem Amazon S3

    AzureBLOB

    wasb fs.azurenative.NativeAzureFileSystem Azure BLOB

  • 22

    BLOB Parcel

  • 23

    HBase Storm

  • 25

    Dryad

    DryadLINQ

  • 26

    GitHub

    https://github.com/MicrosoftResearch/Dryad

  • 27

    HDInsight YARN

  • 28

  • 29

    Naiad

    http://research.microsoft.com/en-us/projects/naiad/

    Dryad DryadLINQ Naiad

    HDInsight

  • 30

    SOSP 2013

    http://research.microsoft.com/pubs/201100/naiad_sosp2013.pdf

  • 31

    Dryad Naiad

    Dryad + DryadLINQ

    Naiad

    API

  • 33

    Cloudera

  • 34

    Azure Marketplace Cloudera Enterprise

  • 35

    D14 Cloudera

  • +