Download - Database story by DevOps
![Page 1: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/1.jpg)
A Database story by DevOpsexploring SQL Database in Production.
![Page 2: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/2.jpg)
Anton Martynenko
Dev & QA & OPS = DevOpsHead of Technical Operations at Youscan
.Net, Windows Azure, Sql Server, NoSql…
Twitter: @[email protected]
![Page 3: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/3.jpg)
YouScan is a leader of the emerging social media monitoring & analytics market in Russia and Ukraine.
Our products:http://youscan.ruhttp://leadscanner.ru
![Page 4: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/4.jpg)
The Reality:
Time is limited
Talk covers a lot of stuff
No time to go into details
If you have a question – make notes and ask later
![Page 5: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/5.jpg)
The Production
Main DB
WWWAndroid
appService iOs app
![Page 6: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/6.jpg)
The Production with more databases
Main DBClientsTools
WWWAndroid
appService iOs app
![Page 7: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/7.jpg)
The Production
Main DB
WWWAndroid
appService iOs app
![Page 8: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/8.jpg)
Database as a Black Box
WWWAndroid
appService iOs app
![Page 9: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/9.jpg)
Database as a source of problems
Main DB
WWWAndroid
appService iOs app
![Page 10: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/10.jpg)
![Page 11: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/11.jpg)
Main DB
Firewall for production DB
WWWAndroid
appService iOs app
![Page 12: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/12.jpg)
DevOps – Developer with access to Prod
Main DB
WWWAndroid
appService iOs app
![Page 13: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/13.jpg)
Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB
![Page 14: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/14.jpg)
Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB
![Page 15: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/15.jpg)
Performance issues
Memory
Disk I/O bottlenecks
High CPU usage
![Page 16: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/16.jpg)
Know your Production hardware!
- Memory
- Disk I/O
- CPU usage
![Page 17: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/17.jpg)
Understand your load
![Page 18: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/18.jpg)
Latencies are important
![Page 19: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/19.jpg)
Understand Database “Math”
Tables
Indexes
Queries
![Page 20: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/20.jpg)
Understand Transaction Isolation
Serializable
Repeatable reads
Read committed
Read uncommitted
![Page 21: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/21.jpg)
Performance: User Experience
WWW
“Place order”
“Success”
Main DB
![Page 22: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/22.jpg)
User’s Transaction is complex
WWW
“Place order”
“Success”
Main DB
![Page 23: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/23.jpg)
User’s Transaction is complex
WWW
“Place order”
“Success”
Main DB
So Slow!
![Page 24: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/24.jpg)
Performance: TOP I/O Queries
![Page 25: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/25.jpg)
Performance: TOP CPU Queries
![Page 26: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/26.jpg)
Google to find TOP IO/CPU Queries
![Page 27: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/27.jpg)
Catch the bottleneck
CPU Score Query
13543541 SELECT * FROM Products WHERE …
7545314 SELECT * FROM Users where …
567541 INSERT INTO Reviews …
IO Score Query
68743513 SELECT * FROM Authors INNER JOIN...
1475214 INSERT INTO OrderQueue …
1246876 INSERT INTO SystemLogs …
![Page 28: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/28.jpg)
Analyze Top IO Queries
Analyze Top CPU Queries
![Page 29: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/29.jpg)
Understand your query
![Page 30: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/30.jpg)
Understand indexes used by query
![Page 31: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/31.jpg)
Understand execution plan
![Page 32: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/32.jpg)
Slow “SELECT”
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
![Page 33: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/33.jpg)
Slow “INSERT”, “UPDATE”
Too many Indexes
Suboptimal Query
Low disk IOPS (or too many inserts)
![Page 34: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/34.jpg)
Understand the query
![Page 35: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/35.jpg)
Catch!
![Page 36: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/36.jpg)
Where is the problem?
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
![Page 37: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/37.jpg)
Root cause: missing index
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Review existing indexes Add new index Or change existing
index
![Page 38: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/38.jpg)
Root cause: query
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Understand execution plan Use Sql query parameters Do select/update minimal
field set
![Page 39: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/39.jpg)
Root cause: low memory
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Review fields in index Archive old data Check DB Engine config Add more memory
![Page 40: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/40.jpg)
Root cause: low IOPS
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Use partitioning Go to SSD Get rid of queries
![Page 41: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/41.jpg)
Root cause: too many indexes
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes Remove unused Remove features
![Page 42: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/42.jpg)
Many indexes = many features
Remove uselessfeatures
![Page 43: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/43.jpg)
Analyze user behavior
![Page 44: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/44.jpg)
Big Table
![Page 45: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/45.jpg)
Table Partitioning
Jan 2014
Feb 2014
Mar 2014
Apr 2014
May 2014
June 2014
Table Orders, All Data
Partitioned table Orders, Partition = Month
![Page 46: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/46.jpg)
Database Sharding Multiple instances,Shard per country
Main DB
GB Main DB
DE Main DB
FR Main DB
Single instance,All Data
![Page 47: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/47.jpg)
Multitenancy Multiple instances,Shard per client
Main DB
Coca Cola Main DB
BMWMain DB
ShellMain DB
Single instance,All Data
NestleMain DB
Macdonald’sMain DB
SiemensMain DB
![Page 48: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/48.jpg)
DB License very expensive
Coca Cola Main DB
BMWMain DB
ShellMain DB
NestleMain DB
Macdonald’sMain DB
SiemensMain DB
![Page 49: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/49.jpg)
Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB
![Page 50: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/50.jpg)
Databases do a few things really well:
• They store and retrieve data
• They enforce relationships between data entities
• They provide the means to query the data for answers
![Page 51: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/51.jpg)
T-SQL, PL-SQL, Whatever-SQL are poor as programming languages
![Page 52: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/52.jpg)
SQL programming tools sux
![Page 53: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/53.jpg)
Debugging and testing is not easy…
Typical T-SQL or PL-SQL mess in Database
![Page 54: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/54.jpg)
What about Refucktoring your SQL code?
Typical T-SQL or PL-SQL mess in Database
![Page 55: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/55.jpg)
Java/C#/etc. with modern tools
Typical T-SQL or PL-SQL mess in Database
![Page 56: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/56.jpg)
WTF your code is doing in Database?
![Page 57: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/57.jpg)
There are no reasons to put your Business Logic
in Database
![Page 58: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/58.jpg)
Data -> Database
Data access -> DAL
Transaction -> (not sql)Code
![Page 59: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/59.jpg)
Deadlocks
![Page 60: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/60.jpg)
Complex code --> Deadlocks
![Page 61: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/61.jpg)
ORM
ORM
Main DB
Application
![Page 62: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/62.jpg)
In Legacy systems more often than notORM adds complexity to the system
![Page 63: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/63.jpg)
ORM
ORM is magic
?
Main DB
Application
![Page 64: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/64.jpg)
Main DB
Lightweight ORM
Application
![Page 65: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/65.jpg)
Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB
![Page 66: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/66.jpg)
![Page 67: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/67.jpg)
Manage schema changes
![Page 68: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/68.jpg)
Change = Script
![Page 69: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/69.jpg)
Two-staged schema changes
![Page 70: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/70.jpg)
Antipattern: Configuration in DB
WWWAndroid
appService iOs app
![Page 71: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/71.jpg)
Antipattern: Lookup tables in DB
![Page 72: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/72.jpg)
Antipattern: Queuing in Database
![Page 73: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/73.jpg)
Keep Schema clean and simple
Get rid of configuration in DB
Get rid of lookup tables in DB
Get rid of write contention
Long term goals:
![Page 74: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/74.jpg)
Database size
![Page 75: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/75.jpg)
Restore/backup
10 minutes
1 Hour
12 Hours
X 3 days
![Page 76: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/76.jpg)
Do Archive old data
![Page 77: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/77.jpg)
Trust is foundation for everything
![Page 78: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/78.jpg)
Thanks! Questions?
![Page 79: Database story by DevOps](https://reader034.vdocuments.mx/reader034/viewer/2022052602/55a20c881a28abe74e8b46f3/html5/thumbnails/79.jpg)
Me: https://www.linkedin.com/pub/anton-martynenko/10/289/9b5
YouScan - http://youscan.ru/LeadScanner - http://leadscanner.ru/
Monitoring & Metrics:Zabbix - http://www.zabbix.com/KissMetrics - https://www.kissmetrics.com/Kibana - http://www.elasticsearch.org/overview/kibana/