service production from d3 pitfall viewpoint
Post on 15-Jan-2015
166 Views
Preview:
DESCRIPTION
TRANSCRIPT
SERVICE PRODUCTION
Walter Liu
2012/06/11
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.1
About me• Architect in Core Tech WRS
• Trend Micro 2007~ Now
• Chief Technical Director in Netgame Dep.• Softstar Inc. Taiwan 1998~2007
• Expertise:• Backend service development and operation
It’s coming ……
The beginning
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.4
Assault - Error 12 !!!
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.5
The nightmare Error 37
Out of stock in Taiwan D3 packages
• D3 packages are out of stock in largest 3 convenient stores in Taiwan and all game shops.
• Some people shared they finally got packages in Jibei island.
• Many people shared they pursued after trucks of convenient stores to get the packages.
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.7
Out of Stock of Taiwan Game Card
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.8
Blizzard Korean lower priority of Taiwan IP
Korean IP
Taiwan IP
Questions?
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.11
What’s the most important things to users?
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.12
What mistakes Blizzard make in this D3 service production?
Why Quantity Estimation?
Cost EffectiveHow?
What if it is wrong?
Scalability & Elasticity
Horizontal Infrastructure
Fail in Scalability
0 1 2 3 4 5 6 7 8 9 10 110
200
400
600
800
1000
1200
Performance
EPIC Fail in Scalability
0 1 2 3 4 5 6 7 8 9 100
200
400
600
800
1000
1200
Performance
Ideal Horizontal Scalability
0 1 2 3 4 5 6 7 8 9 10 110
500
1000
1500
2000
2500
Performance
Elasticity - Unpredictable Traffic
Elasticity - Cloud Solution
Elasticity – Others• Elastic Application Architecture.• Several flexible hardware providers.• Flexible ISPs and pricing.• ……
Customer Service and Social Communication
• Bz is doing pretty bad.• FB Event: Closed-beta account in Taiwan.• Build the image to help their users. Like,
• No explanation about incidents.
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.21
VS.
Incident happens
- Especially your service goes production.
Avengers Assemble !!!
Fast and Responsive Organization
Teams
AwarenessEffective Communication
Fast and Responsive Process• Incident management• Problem management
Some other practices• Interlock with related teams at beginning/middle.
• Customer service prepares resources for burst incoming calls/tickets.
• Customer service prepares training for the new service/product/• Data Center team gives out advices and plan for your project.
• Recruit a Service Manager• Fail case: something that not belong to any team.• Have someone responsible for whole service.
• Update/patch/change SOP• Fail case: Service changed, but your CS don’t know about it. Your
customers are confused when they call your CS.• Fail case: Service changed and caused some trouble, but your
service manager said he didn’t decide/say it.
Not related to these Diablo 3 failures, but important to any system• Availability• Security• Easy to administrate
• System Health/Statistics Monitoring• Easy Deployment• Easy Configure
• Risk Management
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.27
工商服務時間• Web Reputation Service
• Parental Control and Productivity Control• 像是 Hinet色情守門員,防毒軟體的孩童防護鎖• 或者公司不希望員工上班看色情、賭博等網站
• Web Threat Protection• 防止惡意網頁如病毒 /木馬下載、釣魚網站等等。
• Advanced Persistent Threat• 鎖定特定目標• 假冒信件或者其他• 低調且緩慢• 客製化惡意元件• 安裝遠端控制工具• 傳送情資
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.28
WRS – Parental Control
WRS – Web Threat Protection
Funny Diablo 3 sales on Taobao, China
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.31
Thank You!
Risk Management• Identify Critical Failure• Develop a feasible plan to stabilize customer’s
satisfaction.• Workaround.• Rollback.
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.33
Quantity Estimation• Goal: Cost Effective Quantity Estimation
• For estimating• For wrong estimation
• What if the estimation is not correct?• Too few• Too many
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.34
Scalability & Elasticity• Scalability
• Is your application horizontal scalable?
• Elasticity• Speed of commissioning / decommissioning• Max amount of resource can be brought in• Granularity of usage accounting
• Develop the plan for high traffic.
04/10/2023Confidential | Copyright 2012 Trend
Micro Inc.35
Dare to fail• Fail is inevitable for fast changing application. (Web,
service)• Fast changing• Create dare-to-fail process and environment
• Facebook• Backup plan• Rollback plan
top related