Download - Gearman work queue in php
Gearman work queue in php用 Gearman打造 PHP排程
2013.10.05
@appleboy
2
Who am I
Bo-Yi Wu @appleboyhttp://blog.wu-boy.comhttps://github.com/appleboy
任職於瑞昱半導體 RealTek(IC Design House)
3
今日來聊聊排程系統
4
排程使用時機 ?
5
不需要即時回覆都可以使用排程
6
哪些應用需要排程 ?
7
使用者註冊系統
8
使用者註冊
填寫表單完成
寄發認證信
使用者收到回覆
未使用排程
處理資料
透過第三方寄信Amazone SES
9
缺點 Response time過長使用者不需要等伺服器寄信
(來自知名插畫家彎彎 )
10
使用者註冊
填寫表單完成
寄發認證信
使用者收到回覆
使用排程
處理資料
寫入排程
11
Email Queue?Github Social Coding
12
Application Server
Database Database Database
Work Queue
13
執行排程Linux crontab
Ref: http://www.xenstreet.com/2013/03/choosing-the-right-linux/
14
<?php $link = mysqli_connect("host", "user", "passw", "db"); $query = 'SELECT * FROM jobs where status = "on"'; $result = $link->query($query); while($row = mysqli_fecth_array($result)) { // ….... // do work // ….... $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); }
15
每分鐘執行一次*/1 * * * * root cron.sh
16
優點
● 容易瞭解
● 任何程式語言都可以實現
● 快速 Deploy到任何機器
17
缺點
● 每分鐘只能執行一次
● Race condition● 浪費 Database資源
18
換個寫法來試試看
19
每 10秒執行一次
<?php$link = mysqli_connect("myhost","myuser","mypassw","mybd");$query = 'SELECT * FROM jobs where status = "on"';$done = false;while(!$done) { $result = $link->query($query); if ($result->num_rows > 0) { // do work $sql = 'update jobs set status = "off" where id = {$row["id"]}'; $link->query($sql); } else { sleep(10); }}
20
缺點
● 真的 10秒執行一次 ?● Race condition● 更浪費 Database資源
21
如果沒有 Race Condition
22
#!/bin/sh(sleep 5 && /path/script.sh) &(sleep 15 && /path/script.sh) &(sleep 25 && /path/script.sh) &(sleep 35 && /path/script.sh) &(sleep 45 && /path/script.sh) &(sleep 55 && /path/script.sh) &
每 10秒執行一次
23
如何分配處理這些問題 ?
24
Using Gearman
25
What is Gearman?
It allows you to do work in parallel, to load balance processing, and to call
functions between languages.
26
Client inPerl,Python,PHPJava,C#(.Net)
27
evenMySQL,PostgreSQL
28
How Does Gearman Work?
29
Gearman優點
30
Work Queues● 淺顯易懂
Got job do job repeat→ →
Application
Gearmand
Worker
Got job
Do job
31
Work Queues● 淺顯易懂
Get job do job repeat→ →
● 跨語言
任何平台只要能存入字串即可
32
Multiple Language
Application
Gearmand
Application
Worker Worker
33
Work Queues● 淺顯易懂
Get job do job repeat→ →
● 跨語言
任何平台只要能存入字串即可
● 跨平台
當 Client寫入工作時,Worker不需要啟動
34
跨平台
Application
Gearmand
Application
Gearmand
35
Work Queues● 淺顯易懂
Get job do job repeat→ →
● 跨語言
任何平台只要能存入字串即可
● 跨平台
當 Client寫入工作時,Worker不需要啟動● 即時性
可以為同步或非同步
36
Gearman Architecture佈署架構
37
Application
Gearmand
Job
38
Application
Gearmand
Job
Worker
Job
39
Application
Gearmand
Job
Worker
Job Response
40
Application
Gearmand
Job
Worker
Job Response
Response
41
Application
Gearmand
Worker
Worker
Worker
42
Application
Gearmand
Worker
Worker
Gearmand
43
Application
Gearmand
Worker
Worker
Gearmand
Application
44
結論 :Gearman擴充性佳
45
不論要幾台 gearmand跟數個 worker都可以
46
Application只會跟Gearmand溝通
47
多個 Application不管丟出多少個 Jobs
48
最終由多個 Job Server分配工作
49
Synchronous vs
Asynchronous
50
Synchronous Application
Gearmand
Job
Worker
Job Response
Response
51
AsynchronousApplication
Gearmand
Job
Worker
Job Done!
52
Installation安裝方式
53
Installation
● Job Server (C)● Client library (PHP,Python ...)● Worker library (PHP,Python ...)
54
Job Server安裝方式
55
Ubuntu/Debianaptitude -y install gearman gearman-job-server libgearman-dev libdrizzle0
56
CentOS/Fedorayum -y install libgearman gearmand libgearman-devel libdrizzle
58
Gearmand enabled usingMySQL(Mariadb)
PostgresqlSQL Lite
Memcached
59
MySQL Example/etc/default/gearman-job-server
-q mysql --mysql-host=localhost
--mysql-user=xx --mysql-password=xx
--mysql-db=ovoq --mysql-table=gearman_queue
60
Client Librarypecl install channel://pecl.php.net/gearman-1.1.2echo 'extension=gearman.so' > /path/gearman.ini
61
Quick Testphp -i | grep 'gearman'
62
Gearman in PHP Client
63
PHP Client# Create a client object
$gm = new GearmanClient();
# Add default server, default is localhost:4730
$gm->addServer();
# Do Job
$result = $gm->doNormal("reverse", "Hello!");
echo "Success: $result\n";
64
Do background Job$job = $gm->doBackground("reverse", "apple");
65
Gearman in PHP Worker
66
PHP Worker
# Define worker job function
function reverse_fn($job) { //.... }
# fetch message from client
function reverse_fn($job) {
$data = $job->workload();
}
67
PHP Worker# Create our worker object.
$gm= new GearmanWorker();
# Add default server (localhost).
$gm->addServer();
# Register function "reverse"
$gm->addFunction("reverse", "reverse_fn");
# Waiting for job...
while($gm->work());
68
Multiple Jobs$gm->addFunction("reverse", "reverse_fn");
$gm->addFunction("resize", "resize_fn");$gm->addFunction("queue", "queue_fn");
…...
69
Anonymous function$gm->addFunction("reverse", function($job){
$data = $job->workload(); });
70
Gearman in CodeIgniter
71
CodeIgniter Gearman Libraryhttps://github.com/appleboy/CodeIgniter-Gearman-Library
72
Installation via Sparkphp tools/spark install -v1.0.1 codeigniter-gearman
73
Setup Gearmand Server and Portconfig/gearman.php
$config['gearman_server'] = array('localhost'); $config['gearman_port'] = array('4730');
74
CodeIgniter Client
75
Implement Image Resize Jobcreate app controller
76
class App extends CI_Controller{
public function resize_sync()public function resize_async()
}
77
public function resize_sync(){ $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); do {
$this->lib_gearman->do_job_foreground('resize', $data); switch($client->returnCode()) { //.... }
} while ($client->returnCode() != GREARMAN_SUCCESS);}
78
class App extends CI_Controller{
public function resize_sync()public function resize_async()
}
79
public function resize_async(){ $data = json_encode(array('file_name' => 'test.png')); $client = $this->lib_gearman->gearman_client(); $this->lib_gearman->do_job_background('resize', $data));}
80
CodeIgniter Workercreate cli controller
81
class Cli extends CI_Controller{
public static function image_resize()public static function email_queue()public function worker()
}
82
public static function image_resize($job){ $data = json_decode($job->workload(), true); $ci =& get_instance(); $ci->image_lib->initialize($config); $ci->image_lib->resize(); $ci->image_lib->clear();}
83
Define Worker
84
public function worker(){ $worker = $this->lib_gearman->gearman_worker(); $this->lib_gearman ->add_worker_function('resize', 'Cli::image_resize'); $this->lib_gearman ->add_worker_function('queue', 'Cli::email_queue'); while ($this->lib_gearman->work());}
85
Start Worker$ php index.php cli worker
86
Running workers via script#!/bin/shscript="index.php cli worker"folder="/script_path/"max_worker="5"php_bin="/usr/bin/php"count=0current_count=$(ps aux | grep "index.php cli worker" | grep -v "grep" | wc -l)
if test $current_count -lt $max_workerthen
cd $folder$php_bin $folder$script &
elseecho "The worker count has been limited."
fihttps://gist.github.com/appleboy/6806323
87
About Job DataDon't pass 2GB object.
Gearman is not data center.
88
Keep your message data smallAmazone SQS message siez: 256K
http://goo.gl/0MBq8W
89
Scalability
90
Client Client
Job Server
Client
ClientWorker Worker Worker Worker
Job Server
Client
91
you can scale out your clients and workers
as needed
92
Client SettingClient1, Client2: Gearman1Client3, Client4: Gearman2
…...
93
Worker SettingWorker[1-4]:Server[1-2]
…...
94
Add new Job Server...
95
Reconfigure all setting(client and worker program)
96
Reconfigure all setting(client and worker program)
evenDeploy Tool
98
How to improve your architecture?
99
Client Client
Job Server
Client
ClientWorker Worker Worker Worker
Job Server
Client
Proxy (Haproxy ...)
100
Live DemoEmail Queue.
101
Any Questions?
102
Thanks全體工作人員和聽眾