0 to 60 in 3_1 presentation
TRANSCRIPT
-
8/8/2019 0 to 60 in 3_1 Presentation
1/21
Presented by,
MySQL AB & OReilly Media, Inc.
0 to 60 in 3.1
Tyler Carlton
Cory Sessions
-
8/8/2019 0 to 60 in 3_1 Presentation
2/21
Presented by,
MySQL AB & OReilly Media, Inc.
-
8/8/2019 0 to 60 in 3_1 Presentation
3/21
The Project
Medium sized demographics data mining
project
1,700,000+ User base
Hundreds of data points per user
-
8/8/2019 0 to 60 in 3_1 Presentation
4/21
Legacy System Why Upgrade?
+
Main DB (External Users)
Offline backup (Internal Users)
Weekly manual copy backups
Max of 3 simultaneous data
pulls
8hr+ data pull times forcomplex data pulls
Random index corruption
-
8/8/2019 0 to 60 in 3_1 Presentation
5/21
Notes:
Smaller is Better
On average, CPU usage with MySQL was
20% lower than our old database solution.
-
8/8/2019 0 to 60 in 3_1 Presentation
6/21
Why We Chose MySQL Cluster
Scalable
Distributed processing
5 9s Reliability
Instant data availability
between internal
& external users
-
8/8/2019 0 to 60 in 3_1 Presentation
7/21
8 Node NDB cluster
Dual Core 2 Quad 1.8 ghz
16 Gig ram (Data memory)
6x Raid 10 SAS 15k RPM drives
What We Built NDB Data Nodes
-
8/8/2019 0 to 60 in 3_1 Presentation
8/21
What We Built API & MGMT Nodes
3 API nodes + 1 management node
Dual Core 2 Quad 1.8 ghz
8 Gig ram
300 gig 7200rpm (Raid 0)
-
8/8/2019 0 to 60 in 3_1 Presentation
9/21
NDB Issues with a Large Data Set
NDB load times
Loading from backup: ~ 1 hour
Restarting NDB nodes: ~ 1 hour
Note: Load times differ depending on your data size
-
8/8/2019 0 to 60 in 3_1 Presentation
10/21
NDB Issues with a Large Data Set
Indexing Issues
Force index (NDB picks wrong)
Index creation/modification order matters (Seriously!)
Local Checkpoint TuningTimeBetweenLocalCheckpoints - 20 means 4MB (4
220) of write operations
NoOfFragmentLogFiles No. of 4 x 16MB files
None deleted until 3 local checkpoints
On startup: Local checkpoint buffers would overrun
RTFM (two, maybe three times)
-
8/8/2019 0 to 60 in 3_1 Presentation
11/21
NDB Network Issues
Network transport packet size
Buffer would fill and overrun
This caused nodes to miss their heartbeats and drop
This would happen when:
A backup was running
A local checkpoint was running at the same time
Solved by : Increasing network packet buffer
-
8/8/2019 0 to 60 in 3_1 Presentation
12/21
Issues - IN Statements
IN statements die with
engine_condition_pushdown=ON with a set of
apx. 10,000 or more. (caused with zip codes)
Really need engine_condition_pushdown=ON,
but this broke it for us, so we had to disable it.
-
8/8/2019 0 to 60 in 3_1 Presentation
13/21
Structuring Apps: Redundency
Redundant power supply + dual power sources
Port trunking w/ redundant Gig-E switches
# NDB Replicas: 2 (2x4 setup) 64 gig max
data size
MySQL (API Nodes ) Heads: Load balanced
with automatic fail over
-
8/8/2019 0 to 60 in 3_1 Presentation
14/21
Structuring Apps: Internal Apps
Ultimate goal: Offload the data intensive
processing to the MySQL nodes
-
8/8/2019 0 to 60 in 3_1 Presentation
15/21
The Good Stuff: Stats!
Queries per Second (over 20 days)
Average 1100-1500 Queries / Sec
during our peak times
Average 250 Queries / Sec
-
8/8/2019 0 to 60 in 3_1 Presentation
16/21
Website Traffic
Stats for March 2008
-
8/8/2019 0 to 60 in 3_1 Presentation
17/21
Net Usage: NDB Node
All NDB data nodes have nearly identical
network bandwidth usage
MySQL ( API ) Nodes
use about 9 MBs maxunder our current
structure
Totaling 75 MBsduring peak(600 Mbs)
-
8/8/2019 0 to 60 in 3_1 Presentation
18/21
Monitoring & Maintenance
SNMP Monitoring: CPU, Network, Memory, Load, Disk
Cron Scripts:
Node status &Node down notification
Backups
Database maintenance routines
MySQL Clustering book provided the base the scripts
-
8/8/2019 0 to 60 in 3_1 Presentation
19/21
Dolphin NIC Testing
4 node test cluster
4 x overall performance
Brand new patch to
handle automatic Ethernet
failover / Dolphin Fail Over
( beta as of March 28 )
Net Usage: Next steps
-
8/8/2019 0 to 60 in 3_1 Presentation
20/21
Questions?
-
8/8/2019 0 to 60 in 3_1 Presentation
21/21
Contact Information
Tyler Carlton
www.qdial.com
Cory Sessions
CorySessions.comOrangeSoda.com