nos 3 he thong file phan tan google
DESCRIPTION
Hệ thống file phân tán của GoogleTRANSCRIPT
-
5/27/2018 Nos 3 He thong file phan tan Google
1/20
H iu hnh mngnng cao
Ging vin: Hong Xun Du
Email: [email protected]
Khoa Cng ngh thng tin 1Hc vin Cng ngh BC-VT
-
5/27/2018 Nos 3 He thong file phan tan Google
2/20HH mng nng cao Google File System 2
Gii thiu h thng file
phn tn ca GoogleGFS - Google File Systemhttp://research.google.com/archive/gfs.html
-
5/27/2018 Nos 3 He thong file phan tan Google
3/20HH mng nng cao Google File System 3
Ni dung
Mc ch pht trin GFS
Kin trc h thng
Hat ng Hiu nng h thng
-
5/27/2018 Nos 3 He thong file phan tan Google
4/20HH mng nng cao Google File System 4
Mc ch pht trin GFS
Phc v nhu cu lu tr v x l d liu vi khilng rt ln ca Google.
Cho php lu tr v x l cc file c dung lngln, c nhiu GB.
Cung cp khng gian tn c lp vi v tr lutr.
H thng GFS c kh nng kt hp hng ngnnt:
Dung lng lu tr ln n hng trm TB. Cho chp hng trm client truy nhp ng thi vi tn
sut cao.
-
5/27/2018 Nos 3 He thong file phan tan Google
5/20HH mng nng cao Google File System 5
Cc gi thit thit k
H thng c xy dng t nhng phncng tng i r tin, nn kh nng gps c l tng i cao.
H thng phi thng xuyn kim tra phthin s c v phc hi sau s c.
Cc files lu trong h thng thng ckch thc ln, t 100MB n hng chcGB/file.
-
5/27/2018 Nos 3 He thong file phan tan Google
6/20HH mng nng cao Google File System 6
Cc gi thit thit k (tip)
Ti h thng cu thnh t hai thao tcchnh:
Cc thao tc c theo dng (streamingreads): thng h thng c mt lng lnd liu, t hng trm KB n 1MB, hoc lnhn.
Cc thao tc c ngu nhin vi khi lngd liu nh.
-
5/27/2018 Nos 3 He thong file phan tan Google
7/20
-
5/27/2018 Nos 3 He thong file phan tan Google
8/20
HH mng nng cao Google File System 8
Kin trc h thng
-
5/27/2018 Nos 3 He thong file phan tan Google
9/20
HH mng nng cao Google File System 9
Kin trc h thng (tip)
Gm mt master v nhiu chunkservers. Master: qun l ton b cc thng tin v h
thng file:
Khng gian tn (namespace) Thng tin kim sot truy nhp nh x t cc files n cc chunks (phn on) V tr hin ti ca cc chunks
Ton b cc thng tin trn c master lutrong b nh -> gim thi gian tm kim. GFS cn khong 64 bytes qun l mt chunk.
-
5/27/2018 Nos 3 He thong file phan tan Google
10/20
HH mng nng cao Google File System 10
Kin trc h thng (tip)
Chui cc chunkservers: lu tr d liu ccfile.
File c chia thnh cc chunks c kch thc c
nh. Mi chunk c nhn dng bng mt tn nhn
dng ton cc (chunk handle), di 64bits.
Kch thc chunk c nh l 64MB.
Mi chunk thng c 2 bn sao th cp (replicas) Master giao tip vi cc chunkservers thng
qua cc thng ip HeartBeat.
-
5/27/2018 Nos 3 He thong file phan tan Google
11/20
HH mng nng cao Google File System 11
Kin trc h thng (tip)
Master c 2 shadow masters lu bnsao d liu h thng. Khi master chnhgp s c, cc shadow master s thay
th tm thi.
Ton b cc thao tc ca master clog vo a cc b v bn sao th cp
phc v cho khi phc khi gp s c.
-
5/27/2018 Nos 3 He thong file phan tan Google
12/20
HH mng nng cao Google File System 12
Cc thao tc trn file
GFS h tr cc thao tc file c s, baogm: to, xo, m, ng v ghi.
Ngoi ra, GFS cn h tr cc thao tc: Snapshot: to copy ca file hoc cy th mc
vi chi ph thp.
Record append (ni ui file): cho php nhiu
clients cng ng thi ghi ni ui file, mvn m bo tnh nguyn t ca mi thao tcghi ca tng client.
-
5/27/2018 Nos 3 He thong file phan tan Google
13/20
HH mng nng cao Google File System 13
iu khin ghi v dng d liu
-
5/27/2018 Nos 3 He thong file phan tan Google
14/20
HH mng nng cao Google File System 14
iu khin ghi v dng d liu
1. Client yu cu master cung cp thng tin v chunkscn truy nhp (v tr cc replicas).
2. Master tm kim v thng bo kt qu cho client.3. Client y cc thng tin cn ghi n tt c cc replicas
ca chunk.4. Khi cc replicas xc nhn nhn c d liu, clientgi yu cu ghi n primary replica.
5. primary replica chn trt t ghi chuyn tip yu cughi n cc replicas cn li.
6. Cc replicas thng bo hon tt thao tc ghi.7. primary replica thng bo li cho client. Nu c liclient s lp li cc bc t 3-7. Nu vn c li clients lp li ton b qu trnh ghi.
-
5/27/2018 Nos 3 He thong file phan tan Google
15/20
HH mng nng cao Google File System 15
Th nghim o hiu nng
Cu hnh GFS th nghim: 1 master 2 master replicas
16 chunkservers 16 clients
Tt c cc my s dng cu hnh PIII
1.4GHz, 2GB RAM, 2x80GB 5400vng/pht HDD.
Switch 100Mb/s full-duplex.
-
5/27/2018 Nos 3 He thong file phan tan Google
16/20
HH mng nng cao Google File System 16
Th nghim o hiu nng (tip)
-
5/27/2018 Nos 3 He thong file phan tan Google
17/20
HH mng nng cao Google File System 17
Th nghim o hiu nng (tip)
-
5/27/2018 Nos 3 He thong file phan tan Google
18/20
HH mng nng cao Google File System 18
Th nghim o hiu nng (tip)
-
5/27/2018 Nos 3 He thong file phan tan Google
19/20
HH mng nng cao Google File System 19
H thng GFS-2 clusters - Cu hnh
-
5/27/2018 Nos 3 He thong file phan tan Google
20/20
HH mng nng cao Google File System 20
H thng GFS-2 clusters - Hiu nng