designing scalable architectures with mysql proxy

Copyright 2008 MySQL AB The World’s Most Popular Open Source Database 1

Designing ScalableArchitectures

with MySQL Proxy

Giuseppe MaxiaMySQL Community Team Leader - Sun Microsystems

John LoehrerData Architect, Gaia Online

Jimmy GuerreroSr Product Marketing Manager - Sun Microsystems, Database Group

Agenda• Proxy concepts• MySQL Proxy architecture• Proxy with a single back-end• Proxy with multiple back-ends

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.

• broken?•missing feature?• not flexible?


traditional waySolving database problems


1. file a bug report



1. file a bug report2. wait



source code

modify

new source code

compile

• Open source waySolving database problems


bring the logic at application

level

• creative (shortsighted) way

Solving database problems


set the logic at server level

(stored routines)

• creative (enlightened) way



set the logic at protocol level

(proxy)

• creative (more enlightened) way


what can you do with MySQL Proxy• create new commands• filter queries (deny specific queries)• collect statistics on usage• implement usage quotas• execute shell commands• create customized logs• implement server-side pivot tables• start/stop a MySQL server remotely• play movies (seriously!)• make coffee (really?)• sharding• load balancing servers

Basic principles

Basic principles

PROXY CORE

connection hook

read query hook

read result hook

function

Lua script

function

functionfunctionfunction

Lua

??Why not ...{Perl ?

PHP?Javascript?[whatever]?

Lua

• SMALL ( < 200 KB)• DESIGNED for

EMBEDDED systems• Widely used (lighttpd)

lighttpd, like MySQL Proxy, was created by Jan Kneschke

Lua

Very popular among game writers

Proxy overview

connect_server

Lua script

read_auth

read_auth_result

read_handshake

read_query

read_query_result

disconnect_client

global context

session contextsession

contextsession contextsession

contextsession contextsession

contextsession context

Proxy overview/usr/local/sbin/mysql-proxy \ --proxy-lua-script=/path/name.lua

IMPORTANT!THE SCRIPT DOES NOT START UNTIL THE FIRST

CLIENT CONNECTION

read_query and read_query_resultclient

MySQL Proxy

SERVER

query

functionread_query

function

read_query_result

query

result

result

if a query is passed directly to the server, its result is NOT evaluated by read_query_result

read_query and read_query_resultclient

MySQL Proxy

SERVER

query

function read_query

function

read_query_result

query queue

query

result

result

only if a query is added to the query queue, its result is evaluated by read_query_result

queryquery

An example: user quotas• You want to limit user access • When the user reaches a given amount of data,

further queries should be rejected• the quota must work across sessions

(code online)http://forge.mysql.com/tools/tool.php?id=111

http://forge.mysql.com/tools/tool.php?id=111

http://forge.mysql.com/tools/tool.php?id=111

User quotas-- some global variables

-- proxy.global.bandwidth-- will sum up the used bytes

proxy.global.bandwidth = proxy.global.bandwidth or {}

-- session_user will identify the user-- throughout the sessionlocal session_user

User quotas-- session_user needs to be initialized-- when the user information is passed-- (during the authentication)

function read_auth( auth ) session_user = auth.username proxy.global.bandwidth[session_user] = proxy.global.bandwidth[session_user] or 0end

BREAK (a handy Lua idiom)-- simple assignment

-- corresponds to:

if a ~= nilthen a = aelse a = 0 end

if a == nilthen a = 0else a = a end

a = a or 0

BREAK (a handy Proxy Lua function )-- returns an error to the client

function error_result (msg) proxy.response = { type = proxy.MYSQLD_PACKET_ERR, errmsg = msg, errcode = 7777, sqlstate = 'X7777', } return proxy.PROXY_SEND_RESULTend

User quotas-- read_query (1)-- checking if the quota has been-- exceededfunction read_query (packet ) if proxy.global.bandwidth[session_user] > 10000 and session_user ~= 'root' then return error_result( 'you have exceeded your query quota') end-- ...

User quotas-- read_query (2)-- adding to the totalizer

-- ... proxy.global.bandwidth[session_user ] = proxy.global.bandwidth[session_user] + packet:len() proxy.queries:append(1, packet ) return proxy.PROXY_SEND_QUERYend

User quotas-- read_query_result (1)-- adding row headers to the totalizer

function read_query_result(inj) local fields = inj.resultset.fields local rows = inj.resultset.rows if fields then for i = 1, #fields do proxy.global.bandwidth[session_user] = proxy.global.bandwidth[session_user] + (fields[i] and fields[i].name:len() or 0) end

User quotas-- read_query_result (2)-- adding rows contents to the totalizer

if rows then for row in rows do for i = 1, #fields do proxy.global.bandwidth[session_user] = proxy.global.bandwidth[session_user] + (row[i] and row[i]:len() or 0) end end end end

User quotas-- read_query_result (3)-- displaying the current bandwidth

print (session_user .. ' -> ' .. proxy.global.bandwidth[session_user])end

User quotas - what the user seesmysql> select repeat('a', 10000);ERROR 7777 (X7777): you have exceeded your query quota

User quotas (VERY advanced)-- you can create another module-- to be loaded at run time-- (this is really advanced)-- and in such module you define-- a SHOW QUOTAS command

User quotas (VERY advanced)-- load_multi



with MySQL Proxy


John LoehrerData Architect, Gaia Online


quick-start guide to mysql muxing

online hangout

for teens

5 million unique users per month

1.3 billion forum entries

How does Gaia use the proxy?

connection multiplexer

turning 7,500 connections ...

into 100 MySQL connections

MySQL Proxy

PHPMySQLServer

100 connections7,500 connections

Why MUX?

a little history ...

Gaia’s data center in 2003

our server farm...

as our site grew ...

some scaling issues

added more servers ...

site explodes

diagnose the problem

GAIA, under the hood

typical LAMP stack

Share-Nothing Architecture

each apache server = 50 connections

Parent Process

httpd

child httpd

child httpd

...

worker thread

worker thread

listener thread

...

50 connections * 150 servers = stampede

close connections each time

setup / tear-down is expensive

connecting to MySQL

connect readchallenge

writeauth

readauth OK


writeauth

readauth OK write query read result


writeauth

readauth OK write query read result close

75% 19% 6%

MySQL Thread spawning

Client

Client MySQL Server

connect

Client MySQL Server

Threadconnect

new

Client MySQL Server

Threadconnect

accept

new

threads stack up

MySQL I/O

listen

accept

listen

accept

close

end program

NEW THREAD

listen

accept

blocking read

close

end program

NEW THREAD

listen

accept

blocking read

close

end program

NEW THREAD

blocking write

data

listen

accept

blocking read

close

close

end program

NEW THREAD

close thread

blocking write

eof data

talking to multiple databases ...

Forum DBUser DBSession DBForum Page

start session

get userdata

get topic

get related users

save session

idle threads

we need a connection pool

ideally ...

Event Queue Thread Pool

Threadallocate thread

process query

free

release the thread after each query

it’s like a limo ...

... vs. a taxi

decouple threads from connections

Libevent’s callback I/O very sweet!

listen

register callback

event dispatch

active descriptor Trigger callback

epoll/kqueue rulez! (proxy uses it)

MySQL 6.0 taking the right steps ...

In the meantime ...

by default, mysql-proxy isn’t a pooler

lua hooks are awesome!

hack some lua scripts ...

goal: recycle back-end connections

keep clients connected

proxy stays transparent

how to twist the proxy into a pooler

connecting: find an idle back-end

client

Proxy

client

Proxy

idle poolconnect ahead

authentication

Client Proxy Back-End Pool

MySQL Server


MySQL Serverconnect


MySQL Serverconnectallocate



com-change-user



com-change-user

challenge



com-change-user

challenge

authenticate



com-change-user

challenge

authenticate

OK



com-change-user

free back-end

challenge

authenticate

OK

run the query

don’t be afraid to let go

Client Proxy

query


queryallocate


MySQL Serverqueryallocate

process query


MySQL Serverqueryallocate

process query

free back-end

connections scale!



John LoehrerData Architect, Gaiaonline




with MySQL Proxy


MySQL software which can route database queries to the appropriate database in a scale-out environment.

• Improves/scales throughput of reads for online applications• Helps customers reduce cost of adding slaves

• Intelligently route reads across slaves• Use database least behind, least loaded• Remove latent slaves from read rotation• Other distribution algorithms

MySQL Load Balancer


MySQL Enterprise Monitor feature that allows users to trace, monitor, and analyze MySQL query activity for specific servers, users, and applications.

• Adaptive “Evil” query collection/tracing• Historical browsing/analysis

• “Needle in a haystack” identification of worst queries

• Worst execution times, # of execs, etc.

SQL code is the #2 cause of performance issues 97% of those surveyed will use this

MySQL Query Analyzer


MySQL Query Analyzer


Next StepsMySQL Proxy Forumhttp://forums.mysql.com/list.php?146

Documentation – MySQL Proxy

http://dev.mysql.com/doc/refman/5.1/en/mysql-proxy.html

Documentation – MySQL Load Balancer

http://dev.mysql.com/doc/refman/5.1/en/load-balancer.html

Beta Testing MySQL Proxy-Enabled Products

[email protected]

MySQL Proxy Online Poll

http://dev.mysql.com/tech-resources/quickpolls/


Questions?


John LoehrerData Architect, Gaiaonline



designing scalable architectures with mysql proxy

Documents