memcached proxy server development and operation

Tatsuhiko Kubobokko@pixiv.com

13年9月20日金曜日

About me

✦ Tatsuhiko Kubo(bokko@pixiv.com)✦ @cubicdaiya(twitter, github)✦ Senior Software Engineer@pixiv Inc.

Recent Work

✦ mruby_nginx_module✦ Embedded mruby into Nginx

✦ ngx_small_light✦ Dynamic Image Transformation for Nginx

✦ Contributions/pull requests to many other projects on github

Recent WorkWEB+DB PRESS Vol.72

□detailed nginx

flexible configuration and

brilliant scalability

□collaborators

@harukasan@semind

～at pixiv～

Before that,

Background

About datastore & caching strategy

at pixiv

Previously(2 years ago...)

・・・・・・

There was a memcached on each application server.

・・・・・・

AP connected through UNIX domain socket.

・・・・・・

Connects through TCP socket, too.(For data synchronization)

rough-and-ready patch for this

Problems

✦ data synchronization is straining✦ TCP connection overhead✦ processing time is directly proprtional the

number of APs

✦ data restoration is painful✦ memcached is not persistent✦ e.g, when AP server goes down.

Migration to KyotoTycoon

✦ KyotoTycoon supports

✦ data persistency.

✦ data expiration

✦ memcached ASCII protocol.✦ We could migrate without modifying application!

After migration

KyotoTycoon overwhelmed！

KyotoTycoon overwhelmed

✦ KyotoTycoon(using memcached protocol plugin) compared with memcached, ✦ Lower performance

Even so, persistency is some good!

Even so, persistency is some good!We wanted it at that time!

Other Problems

✦ We needed

✦ seamless fail-over mechanism

✦ convenient monitoring mechanism

✦ slow query log(like MySQL)

✦ statistics for whole queries

dealing with these problems

✦ Scale out(For example, add servers)✦ This was difficult for us at least at that time...

✦ Scale up(For example, buy expensive servers)

✦ This was more difficult for us.

✦ Use proxy server!

✦ Actually, I developed one in C.

✦ Twemproxy was not released yet.

✦ Scale out(For example, add servers)✦ This was difficult for us at least at that time...

✦ Scale up(For example, buy expensive servers)

✦ This was more difficult for us.

✦ Use proxy server!

✦ Actually, I developed one in C.

✦ Twemproxy was not released yet.

dealing with these problems

Currently

✦ Applications connect to neoagent by UNIX domain socket✦ reduced consumption of local-ports

✦ neoagent keeps persistent TCP connections to KyotoTycoon✦ bypass overhead of TCP connection

neoagent～A Yet Another Memcached Protocol Proxy Server～

neoagent✦ event-driven & multi-threaded server

✦ proxies a subset of memcached protocol

✦ get, set, add, delete, incr, decr, quit

✦ supports following features

✦ connection-pooling

✦ graceful-restart & online-update

✦ convenient monitoring

✦ etc...13年9月20日金曜日

neoagent architecture✦ core

✦ written in C (C99 style)✦ tools(CLI)

✦ written in Python✦ dependencies

✦ libev, json-c, pthread(core)✦ SCons, Sphinx(tools)

neoagent architecture

✦ Single-master and multiple workers✦ Configuration by JSON✦ Multi-threaded✦ Event-driven

Single Master and multiple workers

neoagentprocess 1

neogentprocess 2

neoagent process 3

KyotoTycoon

proxies

Apache(mod_php)

send request

Previously

✦ each neoagent-process was independent✦ neoagent-process was daemonized by

daemontools

neoagentprocess 1 KyotoTycoon

proxies

neoagent processes spawn gradually

proxies

・・・

proxies

neoagentprocess n KyotoTycoon

proxies

・・・

Previously

(ノ`Д´)ノ┻┻

neoagentworker

process 1

neogentworker

process 2

neoagentworker

process 3

KyotoTycoon

proxies

Currentlyneoagentmasterprocess

neoagentworker

process 1

neogentworker

process 2

neoagentworker

process 3

KyotoTycoon

proxies

Apache(mod_php)

send request

Currentlyneoagentmasterprocess

single-master and multiple workers

✦ Master is responsible for✦ controlling worker processes

✦ Workers are responsible for✦ processing user’s requests

✦ Supervisor may control only Master✦ Now we use Monit instead of daemontools

Configuration by JSON

Configuration by JSON{ "ctl" : { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log", }, "environments" : [ { "name" : "env1", "sockpath" : "/var/run/neoagent.sock", "target_server" : "127.0.0.1:11212", "backup_server" : "127.0.0.1:11213", (abbreviated) }, { "name" : "env2", "sockpath" : "/var/run/neoagent2.sock", "target_server" : "127.0.0.1:11214", "backup_server" : "127.0.0.1:11215", (abbreviated) }, ]}

“ctl” block

sockpath socket path for controlling neoagent

logpath log path for master process

"ctl" : { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log",},

This block is for master13年9月20日金曜日

“environments” block

✦ In neoagent, ✦ environment is the configuration for workers.

✦ Each entry in “environments” is the configuration for each worker.

✦ Master manages workers for each of their of environments.

“environments” block

name environment name

sockpath socket path for communicating neoagent’s worker

target_server primary memcached server

backup_server secondary memcached server

conn_max max connections(backlog size)

connpool_max connection pool size

etc...

document is herehttp://cubicdaiya.github.io/neoagent/

Muti-threaded

neoagent’s threads✦ Master-process

✦ sigwait-thread✦ ctl-thread

✦ Worker-process✦ sigwait-thread✦ event-threads✦ support-thread

Master’s threads

✦ sigwait-thread✦ waiting for a signal from ctl-

thread✦ ctl-thread

✦ receiving a instruction from neoctl through ctl-socket

Worker’s threads✦ sigwait-thread

✦ waiting for a signal from ctl-thread

✦ event-threads✦ processing client requests

✦ support-thread✦ health-checking & statictics

Event-driven

Event-threads’ architecture

✦ event-driven✦ non-blocking I/O✦ multi-threaded

✦ request-queuing & thread-pooling

Event processing phases

multithreaded with request-queuing & thread-pooling

event-thread

request1

request2

request3

・・・

dequeue

event-thread(accept)

request-queue

enqueue

processing requests

Usage at pixiv

✦ Building neoagent✦ Monitoring neoagent✦ Controlling neoagent

Building

Building neoagent

with SCons！

✦ Simple & flexible & programable build tool✦ Auto anaylysis of dependencies✦ Build-configuration is written with Python

✦ Suitable for small or medium scale projects

Monitoring

✦ health-checking worker with Nagios & NRPE

✦ monitoring neostat & Munin

neostat$ neostat -s /var/run/neoagent_st.sock -o{ "datetime" : 2013-09-08 23:12:39 "version": "0.8.7", "host": "localhost", "port": "/var/run/neoagent_st.sock", "environment_name": "yapcasia2013", (abbreviated) "current_conn": 5, "available_conn": 20 (abbreviated)}

neostat -> Munin Graph

current connections

$ neostat -s /var/run/neoagent_st.sock -o{ "datetime" : 2013-09-08 23:12:39 "version": "0.8.7", "host": "localhost", "port": "/var/run/neoagent_st.sock", "environment_name": "yapcasia2013", (abbreviated) "current_conn": 5, "available_conn": 20 (abbreviated)}

neostat -> Munin Graph

neostat outputs current status of neoagent. (For example, current connection numbers)

current connections

$ neostat -s /var/run/neoagent_st.sock -o{ "datetime" : 2013-09-08 23:12:39 "version": "0.8.7", "host": "localhost", "port": "/var/run/neoagent_st.sock", "environment_name": "yapcasia2013", (abbreviated) "current_conn": 5, "available_conn": 20 (abbreviated)}

neostat internal

✦ neoagent workers have a suppport-thread.

✦ support-thread returns JSON as against request of neostat.

slowlog for neoagent■ configuration

■ output

"slow_query_sec" : 0.01, # milli-seconds"slow_query_log_path" : "/var/log/neoagent_slowlog.log","slow_query_log_format" : "json","slow_query_log_access_mask" : "0666",

{ ..., "na_to_ts": 0.000006, "na_from_ts": 0.000003, "na_to_client": 0.014320, "querytxt": "get key1" }{ ..., "na_to_ts": 0.000012, "na_from_ts": 0.000004, "na_to_client": 0.011587, "querytxt": "get key2" }{ ..., "na_to_ts": 0.000009, "na_from_ts": 0.000003, "na_to_client": 0.017788, "querytxt": "get key3" }{ ..., "na_to_ts": 0.000009, "na_from_ts": 0.000009, "na_to_client": 0.036082, "querytxt": "get key4" }{ ..., "na_to_ts": 0.000008, "na_from_ts": 0.000008, "na_to_client": 0.010506, "querytxt": "get key5" }{ ..., "na_to_ts": 0.000014, "na_from_ts": 0.000003, "na_to_client": 0.029464, "querytxt": "get key6" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.000004, "na_to_client": 0.012440, "querytxt": "get key7" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.000003, "na_to_client": 0.043834, "querytxt": "get key8" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.050442, "na_to_client": 0.000015, "querytxt": "get key9" }

slowlog for neoagent

na_to_ts taken time for sending request to target server

na_from_ts taken time for receiving response from target server

na_to_client taken time for sending response to client

querytxt request body

{ ..., "na_to_ts": 0.000006, "na_from_ts": 0.000003, "na_to_client": 0.014320, "querytxt": "get key1" }{ ..., "na_to_ts": 0.000012, "na_from_ts": 0.000004, "na_to_client": 0.011587, "querytxt": "get key2" }{ ..., "na_to_ts": 0.000009, "na_from_ts": 0.000003, "na_to_client": 0.017788, "querytxt": "get key3" }{ ..., "na_to_ts": 0.000009, "na_from_ts": 0.000009, "na_to_client": 0.036082, "querytxt": "get key4" }{ ..., "na_to_ts": 0.000008, "na_from_ts": 0.000008, "na_to_client": 0.010506, "querytxt": "get key5" }{ ..., "na_to_ts": 0.000014, "na_from_ts": 0.000003, "na_to_client": 0.029464, "querytxt": "get key6" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.000004, "na_to_client": 0.012440, "querytxt": "get key7" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.000003, "na_to_client": 0.043834, "querytxt": "get key8" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.050442, "na_to_client": 0.000015, "querytxt": "get key9" }

slowlog is sent to MongoDB by Fluentd！13年9月20日金曜日

Monitoring slow queries with Monitoring Web Application!

Controlling neoagent

Controlling with signals

Problem with signals

✦ complicated & easy to mistake✦ must send multiple signals to master

and workers at a time✦ must link worker’s PID to neoagent’s

environment

Controlling with neoctl

neoctl internal

✦ neoctl sends instructions to master

✦ by UNIX domain socket

✦ master manages worker’ PIDs

✦ and the link between each worker’s PID and environment name, too.

✦ master sends signals master-self and workers.

Graceful restart

neoctlneoagentmasterprocess

neoagentworker

Graceful restart

graceful env1 neoagentworker

neoagentworker

through neoagent_ctl.sock

Graceful restart

neoagentworker

SIGUSR2

Graceful restart

neoagentworker

SIGUSR2

worker(env1) exits after processing remaining requests

Graceful restart

neoagentworker

SIGUSR2

SIGCONT

Graceful restart

neoagentworker

SIGUSR2

SIGCONT

neoagentworker

env1(new)

Graceful restart

neoagentworker

env1(new)

Graceful restart

neoagentworker

env1(new)

Online version update

neoagentworker

update

neoagentworker

update

SIGUSR2

neoagentworker

update

SIGUSR2

workers exits after processing remaining requests

neoagentworker

update

SIGUSR2

SIGUSR1

neoctlneoagentmaster

process(old)

neoagentworker

env1(old)

neoagentworker

env2(old)

neoagentworker

env3(old)

update

SIGUSR2

SIGUSR1

neoagentworker

env1(new)

neoagentworker

env2(new)

neoagentworker

env3(new)

neoagentmaster

process(new)

neoctlneoagentworker

env1(old)

neoagentworker

env2(old)

neoagentworker

env3(old)

neoagentworker

env1(new)

neoagentworker

env2(new)

neoagentworker

env3(new)

neoagentmaster

process(new)

neoctl

neoagentworker

env1(new)

neoagentworker

env2(new)

neoagentworker

env3(new)

neoagentmaster

process(new)

neoctlneoagentmaster

process(new)

neoagentworker

env2(new)

neoagentworker

env3(new)

neoagentworker

env1(new)

with Capistrano！13年9月20日金曜日

Summary

✦ Why memcached protocol proxy server is required?

✦ reduced TCP connections

✦ enhance scalability of system

✦ Monitoring is important！

✦ Don’t guess！ Measure！

✦ Create tools for simplifying operation！

Thanks!

memcached proxy server development and operation

Technology

apache proxy server

http proxy server

firewall and proxy server howto -...

ca siteminder® secure proxy server siteminder secure...

ΣΥΣΤΗΜΑΠΑΡΟΧΗΣΠΛΗΡΟΦΟΡΙΩΝ∆ΙΑ∆ΙΚΤΥΟΥ...

linux11 proxy server

proxy server 2.0 microsoft proxy server 2.0. presentation...

an fpga-based in-line accelerator for memcached ·...

membangun proxy server

using couchbase server to address memcached challenges -...

konfigurasi proxy server

การใช้งานreverse proxy โดย squid...

proxy server

openbsd proxy server

memcached server technology2.pdf

proxy installation guide - red hat customer portal · 2017....

teknik komputer dan jaringan smk negeri · pdf fileteknik...

membangun proxy server

firewall nat proxy server administrasi server

advanced proxy server