storage on ec2 (& cassandra), cassandra workshop, berlin buzzwords
TRANSCRIPT
Storage on EC2(& Cassandra)
Tom WilkieCassandra Workshop 8/06/11
Wednesday, 8 June 2011
ACHTUNG!Data only collected over past 5 days
Didn’t repeat experiments (that much)
EC2 is a moving target
ACHTUNG!
Wednesday, 8 June 2011
Consider:
• Ephemeral vs EBS
• ... vs Instance Type
• ... vs RAID level
• ... vs # threads
• (...vs storage engine)
Not considering:
• Cluster Performance
• Internode latency, throughput
• Tuning...
CORRELATED FAILURES.. .
Wednesday, 8 June 2011
m1.large 7.5 GB RAM, 4 CU, 64-bit, ‘High’ IO
m1.xlarge 15 GB RAM, 8 CU, 64-bit, ‘High’ IO
c1.xlarge 7GB RAM, 20 CU, 64-bit, ‘High’ IO
Cassandra 0.7.6, CentOS 5.5, OpenJDK...
Wednesday, 8 June 2011
Ephemeral StorageEphemeral Storage
Wednesday, 8 June 2011
e·phem·er·al
[ih-fem-er-uhl] Show IPA –adjective
1. lasting a very short time; short-lived; transitory: the ephemeral joys of childhood.2. lasting but one day: an ephemeral flower. –noun3. anything short-lived, as certain insects.
Wednesday, 8 June 2011
1 2 3 40
1000
2000
3000
4000
5000
6000
7000
8000
Ephemeral StorageSeek Performance
m1.large, ephemeralm1.xlarge, ephemeralc1.xlarge, ephemeral
# Devices
See
k / s
7000 IOPs from a disk??
http://www.slideshare.net/davegardnerisme/running-cassandra-on-amazon-ec2
Wednesday, 8 June 2011
1 2 3 40
100
200
300
400
500
600
700
800
900
1000
Ephemeral StorageSeek Performance
m1.large, ephemeralm1.xlarge, ephemeralc1.xlarge, ephemeral
# Devices
See
k / s
Wednesday, 8 June 2011
1 2 3 40
50
100
150
200
250
300
350
400
450
500
Ephemeral Throughputm1.xlarge
Write (Raid-0, dd)Read (Raid-0, dd)Write (Random 10MB chunks)Read (Random 10MB chunks)
# Devices
Thro
ughp
ut (M
B/s
)
Wednesday, 8 June 2011
#
# dd if=/dev/zero of=/dev/sdd bs=512k count=20000
...
10485760000 bytes (10 GB) copied, 201.995 seconds, 51.9 MB/s
#
# dd if=/dev/zero of=/dev/sdd bs=512k count=20000
...
10485760000 bytes (10 GB) copied, 80.3673 seconds, 130 MB/s
Wednesday, 8 June 2011
• Max 4 devices per instance
• Data goes away when instance is terminated (or crashes!)
• Suspect there is some sort indirection layer underneath - thin provisioning / dedupe / CoW or something
• Linux software RAID sucks
Wednesday, 8 June 2011
What happens if a bug in your software causes all your nodes to crash?
ie say a memory leak causes an OOM... on all nodes
CORRELATED FAILURES.. .
Wednesday, 8 June 2011
EBS
Wednesday, 8 June 2011
0 5 10 15 20 25 300
500
1000
1500
2000
2500
3000
EBS Seek performance
m1.large, ebsm1.large, ebsc1.xlarge, ebs
# Devices
See
ks /
s
Wednesday, 8 June 2011
1 2 3 4 5 6 7 8 9 100
100
200
300
400
500
600
700
800
900
1000
EBS Random Reads
m1.xlarge, raid-0
12345678910
# Threads
Tota
l See
k / s
Wednesday, 8 June 2011
0 1 2 3 4 5 6 7 8 9 100
100
200
300
400
500
600
700
800
900
1000
EBS Random Reads
m1.xlarge, raid-0
# Devices
Max
see
k / s
Wednesday, 8 June 2011
1 2 3 4 5 6 7 8 9 100
50
100
150
200
250
300
350
400
450
EBS Random Reads
m1.xlarge, raid-0
maxminavg
# Devices
See
ks p
er d
evic
e pe
r sec
ond
Wednesday, 8 June 2011
Read (Random 10MB chunks)
172.5309.67384.33
458.5
Read (Random
10MB chunks)
1 2 3 4 5 6 7 8 9 100
50
100
150
200
250
300
350
EBS Throughputm1.xlarge
Write (Raid-0, dd)Write (Raid-0, dd)Write (Raid-0, dd)Read (Raid-0, dd)Read (Random 10MB chunks)Read (Random 10MB chunks)
# Devices
Thro
ughp
ut (M
B/s
)
Wednesday, 8 June 2011
• Limited to ~100 IOPS per device?
• Or just 10ms latency?
• Seems to scale pretty linearly for random IO
• Sequential IO limited by network bandwidth, independent of # devices
• shared with other network traffic?
• Linux software RAID sucks
Wednesday, 8 June 2011
What happens when EBS breaks?http://storagemojo.com/2011/04/29/amazons-ebs-outage/
http://status.heroku.com/incident/151
CORRELATED FAILURES.. .
Wednesday, 8 June 2011
+
II
???
Wednesday, 8 June 2011
“Use Elastic Block Storage”http://stackoverflow.com/questions/4714879/deploy-cassandra-on-ec2
“Raid 0 EBS drives are the way to go”http://coreyhulen.org/2010/10/03/%EF%BB%BFcassandra-performance-tests-on-ec2/
“we recommend using raid0 ephemeral disks”http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cold-boot-
performance-problems-td5615829.html#a5615889
Wednesday, 8 June 2011
http://coreyhulen.org/2010/10/03/%EF%BB%BFcassandra-performance-tests-on-ec2/
Wednesday, 8 June 2011
http://coreyhulen.org/2010/10/03/%EF%BB%BFcassandra-performance-tests-on-ec2/
Wednesday, 8 June 2011
0
5000
10000
15000
20000
25000
30000
35000
m1.large, ep
hemera
l
m1.xlarge, ep
hemera
l
c1.xlarge, ep
hemera
l
m1.large, eb
s
m1.xlarge, eb
s
c1.xlarge, eb
s
Inse
rts
/ s
Insert Rates by Instance Type
100 threads, batch mutate size 100, values length 10, 1 column per row, 300 million values
Wednesday, 8 June 2011
Wednesday, 8 June 2011
Wednesday, 8 June 2011
0
425
850
1275
1700
m1.xlarge, ephemeral m1.xlarge, ebs
Get
s / s
Get Rates by Instance Type
100 threads, 700 thousand values
Wednesday, 8 June 2011
Wednesday, 8 June 2011
Range Query Rates by Instance Type
Too slow. No results
Wednesday, 8 June 2011
Wednesday, 8 June 2011
TODO• Repeat experiments
• # threads vs # devices for ephemeral
• Repeat experiments
• Cluster performance - scaling, latency, throughput etc
• Repeat experiments
• Strategies for mixed EBS and Ephemeral?
• Repeat experiments
Wednesday, 8 June 2011
$470110 million IOs, 360 GB-months, 560 machine hours
$470Wednesday, 8 June 2011
Questions?http://github.com/acunu
http://bitbucket.org/acunu
http://www.slideshare.net/acunu
Wednesday, 8 June 2011