Download - BuilHigh Performance Weibo Platform-Qcon2011
![Page 1: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/1.jpg)
Build High Performance Weibo Platform
@TimYang
![Page 2: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/2.jpg)
Background
• Weibo introduction
• 140 Chinese characters
• With photo/video/comments/repost chain
![Page 3: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/3.jpg)
Agenda
�����
�
�
��
�
12
3 4
![Page 4: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/4.jpg)
Part 1
海量存储
![Page 5: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/5.jpg)
每天近亿条新记录,大部分记录变更需要即时反映到业务系统(一致性),如何解决?
![Page 6: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/6.jpg)
MySQL
• 适合海量存储,可任意拆分扩展• 可靠
![Page 7: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/7.jpg)
MySQL 拆分策略
• 仅按 id hash的问题
• 多索引,如关系• 二级索引,如 user_timeline index
![Page 8: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/8.jpg)
MySQL 问题
• 单机容量限制、需要持续拆分• 访问速度需求• 用户• 关系,每秒 5 万次
![Page 9: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/9.jpg)
MySQL + cache 问题
• 雪崩• 一致性• 数据类型不够,需要序列化
![Page 10: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/10.jpg)
NoSQL?
• 单机思路• Redis
• MongoDB
• 分布式思路• Cassandra
• HBase
![Page 11: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/11.jpg)
用好一款开源产品的前提条件是深入了解它的定位。
![Page 12: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/12.jpg)
如何定位
• MongoDB
• Redis
• HBase
• Cassandra
![Page 13: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/13.jpg)
Redis
• snapshot/vm/cache disk/aof
• string/hash/list
• Replication
![Page 14: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/14.jpg)
定位
• 需要高速读写访问• 无高准确性需求• 能容忍短期不可用• 有list/set数据结构需求(optional)
![Page 15: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/15.jpg)
Data Structure
• 计算和存储都依赖数据结构• RDBMS => JSON key value => binary
![Page 16: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/16.jpg)
数据结构
• DB
• JSON/XML
• Binary
![Page 17: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/17.jpg)
JSON 烦恼
• DB
• Cache
• Message Queue
• API
![Page 18: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/18.jpg)
Binary Data Structure
• Numeric: varint, from 1 byte
• 字段名称并不传输,而是传输编号• 多语言:Java, C++, Python...
• 编解码高效
![Page 19: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/19.jpg)
Data Structure
小即是美
![Page 20: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/20.jpg)
Case Study
• 500万粉丝如何在内存中有效存储
• 变长• 动态变化
![Page 21: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/21.jpg)
海量存储
• MySQL,可靠的落地存储
• NoSQL,必要的补充
• 海量存储,小即是美
![Page 22: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/22.jpg)
Part 2
实时计算
![Page 23: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/23.jpg)
一位明星用户可能有上百万粉丝,数据如何实时投递到所有用户?
![Page 24: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/24.jpg)
实时性
• 异步处理• RAM化
![Page 25: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/25.jpg)
一致性
• Timeline, 计数器,关系不是同一服务
• e.g. “有5条新评论”,但 comments_timeline 为空
![Page 26: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/26.jpg)
时序问题
One Data Center
Region 1
Region 2
1001 1002 1003 1004 1005
1002 1001 1004 1003 1005
1001 1003 1002 1004 1005
![Page 27: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/27.jpg)
Part 3
平台接口
![Page 28: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/28.jpg)
游戏规则已经发生了变化,产品的竞争转变为平台和app生态圈之间的竞争
![Page 29: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/29.jpg)
接口设计要素
• 安全• 隐私保护• spam
• 易用• 性能
![Page 30: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/30.jpg)
性能
![Page 31: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/31.jpg)
Part 4
服务治理
![Page 32: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/32.jpg)
框架
• 规范及统一行为
![Page 33: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/33.jpg)
性能
![Page 34: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/34.jpg)
诊断
![Page 35: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/35.jpg)
运维
• 容错• 灾难
![Page 36: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/36.jpg)
小结
![Page 37: BuilHigh Performance Weibo Platform-Qcon2011](https://reader037.vdocuments.mx/reader037/viewer/2022103110/54b78eec4a7959db528b49d3/html5/thumbnails/37.jpg)
�����
("
��
��
��
'� �� )��& *%
����MySQL
Redis
Memcached
Timeline
(�
�#
���
� !
�+��
��$� ��