Badoo , Badoo

330 . 3 Pb 3,5 . 80 .

photos1

photos2

photosN

place_idplace_id: 1..5place_id: 6..11place_id: m..nphotos1

photos2

photosN

:

:

! :

! ! :

! ! ! :

bphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

! $/Gbbphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

! $/Gb! bphotos2

bphotosN

Storage Area Network (SAN)bphotos1

! $/Gb! ! (>500 rps per host)bphotos2

bphotosN

Storage Area Network (SAN)bphotos1

:

7*109 reads / 3.5*106 writes per day

dataset :

7*109 reads / 3.5*106 writes per day

dataset , LRU

:

7*109 reads / 3.5*106 writes per day

bphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

photoscache1

photoscacheN

bphotos

bphotos

Local cache

Local cache

proxy_passproxy_storebphotos

Local cache

BufferHot cacheCold cache

proxy_passproxy_storebphotos

Local cache

BufferHot cacheCold cacheAccess log

proxy_passproxy_storebphotos

Local cache

BufferHot cacheCold cacheAccess log

proxy_passproxy_storebphotos

Cache manager daemon

Local cache

BufferHot cacheCold cacheAccess log

proxy_passproxy_storebphotos

Cache manager daemon

Local cache

BufferHot cacheCold cacheAccess log

proxy_passproxy_storebphotos

Cache manager daemon

-> Hot cache

Local cache

BufferHot cacheCold cacheAccess log

proxy_passproxy_storebphotos

Cache manager daemon

-> Hot cache -> Cold cache

Local cache

BufferHot cacheCold cacheAccess log

proxy_passproxy_storebphotos

Cache manager daemon

-> Hot cache -> Cold cache Cold cache

?

photoscache2

photoscache1

photoscache3

? Round-robin?

photoscache2

photoscache1

photoscache3

? Round-robin?Hash % count?

photoscache2

photoscache1

photoscache3

? Round-robin?Hash % count?hash(example_url) = 5server_idx0 = 5 % 3 = 2

photoscache2

photoscache1

photoscache3

? Round-robin?Hash % count?hash(example_url) = 5server_idx0 = 5 % 3 = 2server_idx1 = 5 % 4 = 1

?

photoscache2

photoscache1

photoscache3

Round-robin?Hash % count?

? Round-robin?Hash % count?Consistent hashing?

photoscache2

photoscache1

photoscache3

Consistent hashing

0hash(sharding_key)

Consistent hashing

0hash(sharding_key)

Consistent hashing

0hash(sharding_key)

A

Consistent hashing

0hash(sharding_key)

BA

Consistent hashing

0hash(sharding_key)

BAC

Consistent hashing

0hash(sharding_key)

BAC

Consistent hashing

0hash(sharding_key)

BAC

BAC

photoscache1

bphotos

photoscache2

photoscache3

photoscache4(reserve)

Load balancer

photoscache1

Load balancer

photoscache2

photoscache3

photoscache4(reserve)

bphotos

. Hitrate ( ) 98% 80k 1600 rps bphotos

. Hitrate ( ) 98% 80k 1600 rps bphotos3 (, , )

. Hitrate ( ) 98% 80k 1600 rps bphotos3 (, , )

webp, progressive jpeg resize/crop , (blur, pixelize)

CDN?

CDN?

CDN?

CDN?

CDN?

CDN

. 2013

bphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

photoscache1

photoscacheN

. 2013

bphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

photoscache1

photoscacheN

80 x bphotos = ~560Tb 40 x photoscacheX 2

. 2013

bphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

photoscache1

photoscacheN

1 x POINT OF FAILURE

. 2013

bphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

photoscache1

photoscacheN

2 x POINT OF FAILURE

. 2013

bphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

photoscache1

photoscacheN

3 x POINT OF FAILURE

. 2013

bphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

photoscache1

photoscacheN

3 x POINT OF FAILURE! MAINTENANCE

. 2013

bphotos1

bphotos2

bphotosN

Storage Area Network (SAN)

photoscache1

photoscacheN

3 x POINT OF FAILURE! DATA LOSS! MAINTENANCE

v.1

bphotos

Main partitionLocal FSFiber

Storage Area Network

bphotos

Local FS

Main partition

Backup partitionFiber

Storage Area NetworkFiber

Storage Area Network v.1

Async queue

bphotos

Local FS

Main partition

Backup partitionFiber

Storage Area NetworkFiber

Storage Area Network v.1

Async queue

bphotos

Local FS

Main partition

Backup partitionFiber

Storage Area NetworkFiber

Storage Area Network

Buffer partition v.1

Async queue

bphotos

Local FS

Main partition

Backup partitionFiber

Storage Area NetworkFiber

Storage Area Network

Buffer partition! NO DATA LOSS v.1

Async queue

bphotos

Local FS

Main partition

Backup partitionFiber

Storage Area NetworkFiber

Storage Area Network

Buffer partition! NO DATA LOSS! POINT OF FAILURE v.1

Async queue

bphotos

Local FS

Main partition

Backup partitionFiber

Storage Area NetworkFiber

Storage Area Network

Buffer partition! NO DATA LOSS! POINT OF FAILURE! MAINTENANCE

v.1

Dphotos

Async queue

bphotos

Local FS

Main partition

Backup partitionFiber

Storage Area NetworkFiber

Storage Area Network

Buffer partition

Dphotos

Async queue

bphotos

Local FS

Main partition

Backup partition

Buffer partition

Dphotos

Async queue

bphotos

Local FS

Main partition

Backup partition

Buffer partition

Dphotos

dphotosN

Buffer partition

Main partition

dphotosN+1

Buffer partition

Main partition

Dphotos

dphotosN

Async queue

Buffer partition

Main partition

dphotosN+1

Buffer partition

Main partition

Dphotos. Upload

Async queue

dphotosN

dphotosN+1

Round robin + health checks

Load balancer

Dphotos.

photoscache

Async queue

dphotosN

dphotosN+1

Round robin + health checks

Dphotos.

photoscache

Async queue

dphotosN

dphotosN+1

HIT

Round robin + health checks

Dphotos.

Async queue

dphotosN

dphotosN+1

MISS

Round robin + health checksphotoscache

Dphotos.

Async queue

dphotosN

dphotosN+1

MISS

Round robin + health checksphotoscache

Dphotos.

dphotosN

Async queue

Buffer partition

Main partition

dphotosN+1

Buffer partition

Main partition

Dphotos.

dphotosN

Async queue

Buffer partition

Main partition

dphotosN+1

Buffer partition

Main partition

Dphotos.

dphotosN

Async queue

Buffer partition

Main partition

dphotosN+1

Buffer partition

Main partition

Dphotos.

dphotosN

Async queue

Buffer partition

Main partition

dphotosN+1

Buffer partition

Main partition

Dphotos.

Dphotos.

Dphotos.

Dphotos.

Dphotos. ?

Dphotos. ?

Dphotos. ? 1.5 , SAN

photoscache

photoscache

CDN

photoscache

dphotos

CDN

Storage layer

photoscache

dphotos

CDN

Storage layerLocal drives

photoscache

dphotos

Storage Area Network

CDN

Storage layerLocal drives

?

? http://pinba.org

? http://pinba.org -> ->

? http://pinba.org -> -> Immutable

? http://pinba.org -> -> Immutable Resize

? http://pinba.org -> -> Immutable Resize -

? http://pinba.org -> -> Immutable Resize - - -