gogolook sqs lesson learnt
TRANSCRIPT
Gogolook SQS lesson learnt
Hello!I am Kakashi
You can find me at:Twitter: @kakashiliu
The story is ...
Previous Status
◎ Worker: c3.large * 30~50, spot instance◎ CPU utilization: less than 20%◎ Messages: over than 1 billion / per day
SQS 基本介紹
SQS 基本設定 2
◎ First 1 million SQS requests per month are free◎ $0.476 / per 1 million SQS requests ◎ A single request can have from 1 to 10 messages,
up to maximum total payload of 256 kb◎ requests including:
CreateQueue, ListQueues, DeleteQueue, SendMessage, SendMessageBatch, ReceiveMessage, ChangeMessageVisibility, ChangeMessageVisibilityBatch, DeleteMessage, DeleteMessageBatch, SetQueueAttributes, GetQueueAttributes, GetQueueUrl, AddPermission, and RemovePermission.
SQS新政1 : 流程改善
◎ 每個worker每隔1min, 用crontab叫他們復活, 沒事幹就死掉 => cpu utilization low, cpu loading high
◎ Solution:利用 while True 加上 SQS long polling (20s) , 沒事做的時候sleep 60s
◎ 結果: Loading降低, CPU utilization 沒提升很多
SQS新政2 : 斷開鎖鏈
◎ 發現有network bound: 3rd library loggly 沒有使用keep-alive的方式連接
◎ Solution:1) python lib 替換 urllib2 為 requests.session 2) 並且使用batch sent log
◎ 結果: 清queue速度快一倍 worker cpu utilization 上升數倍
SQS新政3 : 減少request量
◎ 利用sent message batch的方式, Batch sent 100 messages or 塞滿256kb在發送
◎ Solution:patch掉原本送queue的function, 並且利用 atexit 去避免worker被關閉時, 有些message還在 local queue裡面
◎ 結果: sent, receive , delete 量減少99%
SQS新政4 : 調整單機工人數目
◎ Message 多的開多一點 thread去處理
◎ 其中還可以利用 gevent 去避開一些 network bound的問題
Result
◎ Spot instance: 30~50 -> 2◎ CPU utilization: 20% -> 80%◎ Message Counts: 1 billion -> 10million
◎ 處理時間
Result
Cost Reduction around 70%
Bonus: SQS alternative
Fluentd (Before)
Fluentd (After)
Traditional Flow
AP server SQS Worker Mongodb
Big Query
Fluentd Flow
AP server
FluentdForwarder
FluentdStation
Mongodb
Big Query
Fluentd Flow
AP server
FluentdForwarder
FluentdStationELB
FluentdStation
FluentdStation
FluentdStation
