open source world '09 scaling web applications
TRANSCRIPT
-
8/14/2019 Open Source World '09 Scaling Web Applications
1/36
2009-08-12
Scaling Web ApplicationsGavin M. Roy, Chief Technology Officer - myYearbook.com
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
2/36
About myYearbook.com
Social Network, Top Teen Destination
2007
100M Page views per Month
2009 1.5B Page views per Month
Top 20 Page view site as ranked by comScore
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
3/36
Its a series of tubes.Former US Senator Ted Stevens
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
4/36
Early Stages
Emphasis on the application
Usually Monolithic and tightly coupled
Many base architectural decisions are made
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
5/36
Traffic InfluxPlaying whack-a-mole: the price of popularity
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
6/36
Pressure Point: Database
Database Strain
High IO
Slow Queries
Single Point of Failure Maintenance
High Connection Counts
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
7/36
Solutions for Database Scaling
Many methods for scaling databases
Replication
Sharding
Removing Joins Moving data to different servers
Caching
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
8/36
Solving Database IO Problems
Database in Memory
Expensive!
Linux Kernel Disk Buffer Cache
Works Very Well Faster Disks
Direct Attached > Remote Storage (SAN/NAS/etc)
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
9/36
Solving Slow Queries
Programmers are not DBAs
Proper indexing
Remove Complexity by reducing Joins
Data partitioning in OLTP
Good Schema > Fully de-normalized data
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
10/36
Solving Slow Queries
Sharding by User
CRC of Unique Key determines server where data lives
PostgreSQL via pl/proxy
MySQL via client code More complex to back-up
Support partial outages
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
11/36
Database Scaling via Replication
Multi-Master Replication
All servers must agree on data prior to commits (slow).
Consistent data across all servers
Master-Slave Replication
All writes on master server
Reads distributed across slaves
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
12/36
Service Oriented Architecture
Databases separated by application or service
Service data is stand alone from primary data
May need data from primary service
Need tools to keep data relevant
Allows for distributed application architecture
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
13/36
Solving Large Connection Counts
Connection Pooling
Java has this built in via JDBC connection pooling
Most other languages do not
Not Persistent Connections
Maintains Persistent Connections
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
14/36
Database Connection Pooling
PostgreSQL
pgBouncer - Skype
pgPool
MySQL
DBSlayer - NY Times
Cherokee Web Server
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
15/36
Caching
Everyones favorite daemon: memcached
Clients automatically pool and shard data
Key / Value Store
Volatile*
Economy of Scale
Failure of small memcache cluster none is impacting
Many nodes make light workWednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
16/36
Caching Bottleneck: Network
Cache servers can transfer lots of data quickly
Keep regular insight into your network health
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
17/36
Removing Bottlenecks: Caching
Local Caching
Shared Memory (System V Shared Memory / Posix)
PHP - APC / Zend
File-system
memcached on web server?
Multi-tier caching, local caching then memcached cluster caching
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
18/36
Pressure Point: Web Application
Load Balancing
Tightly Coupled
High CPU Utilization
High IO and Shared Filesystem Data
Scaling Across Multiple Servers
Application Deployment
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
19/36
Spreading the Load
Hardware
Expensive
Software
By Service
DNS Round Robin
Relies on Client
Content Delivery NetworkWednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
20/36
Decoupling Your Application
Web Applications spend a lot of time waiting
Database connection time
Database query time - Even writes
Caching service connection time
Enter Message Queues and Remote Workers
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
21/36
Decoupling via Message Queues
Offload data writes and actions
Offers granularity and control of impact
Scale workers to scale transaction rate
Many good brokers and standards
RabbitMQ, ActiveMQ, ZeroMQ, OpenAMQP, Java Message Service
Be language / vendor neutral
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
22/36
Decoupling via Remote Workers
Message Queues
Gearman
Hadoop
Map / Reduce
Offload costly CPU activity to farms of machines
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
23/36
Resolving File IO Slowness
Scale-Out NAS
Facebooks Haystack
PHP: Optcode Caching
Logging, syslog to remote servers
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
24/36
Scaling Across Many Servers
Common Configuration
memcached / tokyo tyrant
NAS stored configuration files
Health Checks
Function of Load Balancers
Need for additional insight
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
25/36
Application Deployment
Pull vs Push
Push works with low qty of hosts
Easy! Use scripts and ssh
Pull upon notification of code updates
Version control
Include an internal API call that returns version information
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
26/36
Knowing Your PipesWhy is the site down?
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
27/36
Clogged Pipes?
CPU Pegged
Database Bloat
Disk Issues
Network Issues
Lots of Traffic
Hardware Problems?
Crashed Daemons
Crashed Servers
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
28/36
Monitor Your Network
Servers
System Profile (CPU, Memory, Disk)
Network Hardware
Bandwidth
System Profile
Daemons
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
29/36
Monitor Your Network
Use Internal Services
Nagios
Hyperic
OpenNMS
Use External Services
Gomez
Alertra
Pingdom
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
30/36
Monitor your Application
Build Profiling into Application
How much time do I spend connecting to a database?
How much time do I spend in business logic?
Sample profile data in Application
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
31/36
Trending Application Behavior
How can I predict future utilization based upon existing growth?
Identify trends in application behaviors
Troubleshoot when something is just not right
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
32/36
StaplrOr whats my entire site up to?
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
33/36
Staplr: PostgreSQL
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
34/36
Staplr
Pollers
Apache HTTP, Lighttpd, Cherokee
SNMP (APC PDUs, Isilon NAS, Net-SNMPD)
Memcache
MySQL
PgBouncer
PostgreSQL
SysStat
and many more
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
35/36
Monitor Your Application
Use client profiling
JavaScript sends statistical data to server
Profile JavaScript action times
Page load times
Abandonment rates
Analytical data
Wednesday, August 12, 2009
-
8/14/2019 Open Source World '09 Scaling Web Applications
36/36
Questions?
Follow me on twitter: http://twitter.com/crad
Contact me via email: [email protected]
http://twitter.com/cradmailto:[email protected]:[email protected]://twitter.com/cradhttp://twitter.com/crad