cloudera...

Download Cloudera Manager4.0¨NameNode-HA‚»ƒƒƒ¼è³‡–™

Post on 24-May-2015

5.189 views

Category:

Technology

5 download

Embed Size (px)

DESCRIPTION

2012年7月のセミナーおよび説明会で使用した、CDH4の高可用NameNode(NameNode-HA)とCloudera Manager4.0の資料です。

TRANSCRIPT

  • 1. CDH

2. HDFS HA Cloudera Manager 3. 4. CDH4 + Cloudera Enterprise 4.0:14IT 2 5 36 5. CDH4 + Cloudera Enterprise 4.0:14IT 2 5 36 6. CDH4 + Cloudera Enterprise 4.0:14IT 2 5 36 7. CDH4 + Cloudera Enterprise 4.0:14IT 2 5 36 8. HDFSHDFS HIGH AVAILABILITY 9. reliability = MTBF/(1 + MTBF) MTBF: 1111 maintainability = 1 / (1 + MTTR) MTTR: availability = MTTF / MTBF MTTF: MTBF = MTTF + MTTR 10. 1020,0003.2919(2009) 0 17001(4PB) 11. () 182522 10.58 HA 8(0.23) 12. NN: 1-2 15 MTTR() Heartbeat + DRBD DN : 11DN 31 13. HA (%) HA 99.993 HA() 99.996 1 HA160 2 2009-2010 14. 15. HDFS HA NN NN 1 DNNN NN NN 16. NN HA NNHA NN : OS NN NN NN 17. DaemonDaemon ZooKeeper editlog () () () DaemonDaemon 18. (NFS) editlog () () ()Daemon Daemon 19. ( ) URINN / NN 20. Zookeeper ZK NN NN NNZKFailoverController (ZKFC) ZKFC NN ZK ZKNN 21. DaemonDaemon ZooKeeper ZK ZK ZKFC ZKFC / / () ()AB 22. HA: NN SPOFHA HAIP tcp,soft, intr, timeo=60, retrans=10 NFSBookKeeper (HDFS-3077) 23. HA: NN NN NN RPCNN() SSHNN kill -9 NFSIP PDUIPSTONITH OS 24. HA: ZK ZK1NN1YARNRM ZK HBaseZK NNNNZKFC ZKFC 25. HA: NN DN NNedits DN NN () 26. HA: /NN /NNHW NN ECC48GB RAID5 27. CLOUDERA MANAGER 4 28. Cloudera Manager(CM) CDH1 CDH 29. WebUI 30. () Hadoop 31. CM CDH DN/RS/NM DN/RS/NM NN DN/RS/NM DN/RS/NM RM DN/RS/NM DN/RS/NM HMaster DN/RS/NM DN/RS/NM Hue DN/RS/NM DN/RS/NM DN/RS/NM DN/RS/NM 32. CM CDH DN/RS/NM DN/RS/NM NN DN/RS/NM DN/RS/NM Cloudera RM DN/RS/NM DN/RS/NM Manager HMaster DN/RS/NM DN/RS/NM Hue DN/RS/NM DN/RS/NM DN/RS/NM DN/RS/NM CM 33. Cloudera Manager PC DN/RS/NM DN/RS/NM web CM DN/RS/NM DN/RS/NM DN/RS/NM DN/RS/NM CM DN/RS/NM DN/RS/NM DN/RS/NM DN/RS/NM DN/RS/NM DN/RS/NM DB DB 34. CM() 35. CM() 36. CM Service Types () : HDFS, MapReduce, HBase Service Instances () : hdfs1, mapred1 Role Types () : DATANODES, JOBTRACKER Role Instances () : datanode1 Commands () : Restart 37. CM4.0 API Ubuntu/Debian PostgreSQL/Oracle Cloudera 38. CM4.0 () (!) Gateway( ) CMLDAP kill (MRStreaming) HDFS 39. CM Free Edition Kerberos 50 ( ) 40. Free Edition Enterprise Edition Free Edi-on Enterprise Edi-on () CDH API 50 -- -- -- DL Cloudera Enterprise 41. CM UI 42. CM4.0 LAN DB PostgreSQL DB DB DB DB CDH Debian/Ubuntu 43. SERVICE AND CONFIGURATION MANAGER 44. SCM Service and Configuration Manager ( ) install everything, everywhere Free Edition Enterprise() 45. 46. HBase graceful stop 47. 2 NN2NN NN1GB 48. Enterprise Only 49. 50. HDFS/MapReduce 11 51. () // CDH 1 52. 2 53. = alternatives /etc zip /etc Gateway 54. () UI zip wget/curl DL 55. 56. HDFS HA HDFS basic HA UI HA, 16 CM 57. 3 HA(1)1. NN 58. 3 HA(2)2. edits3 59. HANN 60. CDH3CDH4 : CMCDH : HDFS 61. CDH3CDH4() CMDB CDH HDFS: HBase, Oozie: 62. DB PostgreSQLOracle CMDB CM3.7: Oracle PostgreSQLSCM DBMySQL PostgreSQL 63. CM /etc /etc XML CMDBCMUI 64. CLOUDERA MANAGER API 65. RESTful CM API REST API ? = URL HTTP(GET, POST, etc) = POST/PUT CM API HTTP JSON Free Edition 66. API 67. :$ curl -u "admin:admin" http://$CM/api/v1/clusters{"items" : [ {"name" : "Cluster 1 - CDH4","version" : "CDH4"}, {"name" : "Cluster 2 - CDH3","version" : "CDH3"} ]} 68. :$ curl -X POST -u admin:admin http://$CM/api/v1/clusters/Cluster%201%20- %20CDH4/services/my_hbase/commands/hbaseCreateRoot { "id" : 142, "name" : "CreateRootDir", "startTime" : "2012-05-06T20:56:57.918Z", "active" : true, "serviceRef" : { "serviceName" : "my_hbase", "clusterName" : "Cluster 1 - CDH4" } } 69. Python API github (Apache License) http://cloudera.github.com/cm_api/ 70. CM API Web UI API iPhone 71. CLOUDRA MANAGER ENTERPRISE EDITION 72. Enterprise Edition CM Free Edition Kerberos 73. Cloudera Manager Free Edi-on Cloudera Manager Ent. Edi-on Yes Yes API Yes Yes HDFS, MapReduce, MR2, HBase, Hue, Oozie, Zookeeper Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes // Yes Yes Yes Yes Yes Yes Free Yes LDAP Yes Kerberos Yes Yes Yes Enterprise Yes Yes Yes Yes Yes Yes 74. HDFS: IO, , etc MapReduce: Map, Reduce 75. ()SCM 76. (HDFS) 77. (HDFS) 78. (HDFS) 79. (MapReduce) 80. IPID CPU 81. () 82. () 83. () 84. () 85. () 86. 87. () MapReduce, Hive, Pig, Oozie map/reducemap MapReduce MR1 88. 89. () 90. () 91. (jobkill)(MRStreaming) 92. () 2 93. () 94. 76 20:0021:00 a,b,c,d mapreduce1 WARN 95. 96. 97. 98. (HDFS) 99. (HBase) 100. 2 CDH 101. (HDFS) 102. 103. etc. MapReduce CSVExcel 104. 105. () 106. () 107. : Cluster Stats zip zip 108. Cluster Stats 109. Cluster Stats 110. 111. HDFS HA HDFSNNSPOF Cloudera Manager 3 Hadoop 112. Cloudera Manager Hadoop 100 Enterprise Edition 113. DEMO 114. (CDH) http://www.youtube.com/watch? v=CobVqNMiqww CDH 115. Thank : You! info-jp@cloudera.com 03(6228)7930 cloudera.co.jp twiSer.com/ ClouderaJP facebook.com/ cloudera