PHP PThread多线程编程之并发发送HTTP请求

前言

之前由于项目需求，采用了PHP 的pthread扩展编写异步并发程序。需求是这样的：单台服务器需要将每秒内上报的战报上传至服务器并保证上报成功率在99.9%以上，由于合作平台极其不稳定，大概率上报超时涉及重发。那么上报就必须使用异步队列，但是进程是很消耗系统资源，PHP本身是不支持线程的，最后采用了PHP 的pthread扩展进行多线程编程。（期间用过swoole所谓协程，测试数据差强人意；想过go重写，开发周期有限）

由于网上关于php多线程编程资料甚少（PHP本身的弱项），加上pthread缺少报错信息，导致调试难度。留下采坑过程，以供借鉴。

环境安装

由于LNMP模式安装的PHP非线程安装版本为php7.1, 为保持保持一致重编译线程版本也保持php7.1。官方提供7.1 pthread版本无法编译成功(Zend语法不兼容)，感谢SKDSKL¹提供的 https://github.com/sjdskl/pthreads-php7.1.git 提供的修改版本可成功编译安装。

git clone  https://github.com/sjdskl/pthreads-php7.1.git
cd pthreads-php7.1
./configure --with-php-config=/usr/local/php/bin/php-config
make && make install
echo "extension=pthreads.so" >> /usr/local/php/etc/php.ini

/usr/local/php/bin/php Benchmark.php

由于LNMP开启线程安全是不值得的，所以同时存在2个版本的php（Thread Safety和No Thread Safety）。

pthread使用注意事项²

1. 线程创建后，不能使用父线程的变量，诸如$GLOBALS或global等用法都无法操作父线程的全局变量

2. 线程类的属性不能直接进行哈希表（数组）操作，如：

//这样是无效的
$this->var1["hello"] = "world"; 
//改为
$this->var1 = ["hello"=>"world"];

因为线程类属性的赋值是通过序列化实现的，其本质是存储了序列化数据。

3. 不能调用父线程的redis、mysql等非标量的成员变量（引用类型的对象，涉及父线程的内存空间），如果是标量就可以。

4. 最佳实践：为了安全起见，建议不要引用父线程的任意变量，用到的参数都通过Thread构造函数传入，可以把php线程当成轻量级的进程，不要做共享内存的操作。

线程类

/**
 * Created by PhpStorm.
 * User: liugaoyun
 * Date: 2018/8/4
 * Time: 上午11:51
 */

namespace console\threads;

class DemoThread extends \Thread
{
     
    public $dbConfig = null;    //db配置

    public $redisConfig = null; //redis配置

    public $debug = false;  //是否是debug模式


    public function __construct($dbConfig, $redisConfig, $debug = false)
    {
     
        $this->dbConfig = (array)$dbConfig;
        $this->redisConfig = (array)$redisConfig;
        $this->debug = $debug;
    }

    public function run() {
     
        //TODO:DB初始化
        $pdo = new \PDO($this->dbConfig['dsn'], $this->dbConfig["username"], $this->dbConfig["password"],array(\PDO::ATTR_ERRMODE => \PDO::ERRMODE_EXCEPTION));
        $pdo->exec("SET NAMES '" . $this->dbConfig["charset"] . "' COLLATE '" . $this->dbConfig["collation"] . "'");
        $pdo->exec("SET CHARACTER SET '" . $this->dbConfig["charset"] . "'");
        $pdo->setAttribute(\PDO::ATTR_DEFAULT_FETCH_MODE, \PDO::FETCH_OBJ);

        //TODO:redis初始化
        $redis = new \Redis();
        $redis->pconnect($this->redisConfig['host'], $this->redisConfig['port']);
        $redis->auth($this->redisConfig['passwd']);

        if(isset($this->redisConfig['options'])){
     
            foreach ($this->redisConfig['options'] as $option=>$value){
     
                $redis->setOption($option, $value);
            }
        }

        //战报批量回传地址
        $url = API_URL . '/xinternal/h5game/record/batch-report.htm';

        $st = millitime();
        $round_num = 0;
        //每间隔3秒发送一次ACK,避免休眠断线
        $last_ack = 0;

        while (millitime() - $st < 300000){
     
            if(time() - $last_ack > 3){
     
                $last_ack = time();
                //TODO:redis和mysql连接状态检测
                try{
     
                    echo "Redis ACK\n";
                    $ping = $redis->Ping();//获取ping标识
                    if(stristr($ping, 'PONG') === false){
     
                        throw new \Exception("Redis Connect Getway", 1000);
                    }
                }catch (\Exception $e){
     
                    //TODO:redis初始化
                    echo "Redis连接断开，重新初始化\n";
                    $redis = new \Redis();
                    $redis->pconnect($this->redisConfig['host'], $this->redisConfig['port']);
                    $redis->auth($this->redisConfig['passwd']);

                    if(isset($this->redisConfig['options'])){
     
                        foreach ($this->redisConfig['options'] as $option=>$value){
     
                            $redis->setOption($option, $value);
                        }
                    }

                    continue;
                }

                try{
     
                    echo "MySQL ACK\n";
                    $sql = "SELECT bl_uuid FROM battle_log LIMIT 1";
                    $pdo->query($sql);
                }catch (\PDOException $e){
     
                    echo "PDO 连接断开，重新初始化\n";
                    $pdo = new \PDO($this->dbConfig['dsn'], $this->dbConfig["username"], $this->dbConfig["password"],array(\PDO::ATTR_ERRMODE => \PDO::ERRMODE_EXCEPTION));
                    $pdo->exec("SET NAMES '" . $this->dbConfig["charset"] . "' COLLATE '" . $this->dbConfig["collation"] . "'");
                    $pdo->exec("SET CHARACTER SET '" . $this->dbConfig["charset"] . "'");
                    $pdo->setAttribute(\PDO::ATTR_DEFAULT_FETCH_MODE, \PDO::FETCH_OBJ);

                    continue;
                }
            }


            //一次获取20条数据
            $battle_list = [];

            for($i = 0; $i < 20; $i++){
     
                $item = $redis->rPop('battleLogCallbackAsyncQueue');
                if(empty($item)){
     
                    break;
                }

                $battle_list[] = $item;
            }



            $request_data = [
                'data' => json_encode($battle_list),
                '_t'=>millitime()
            ];


            $response = $this->httpPost($url, $request_data);

            if($response && $response['code'] == 0){
     //成功处理


            }else{
     //失败处理

            }

            $round_num++;
        }

        exit(0);
    }

    public function httpPost($url, $params) {
     
        $curl = curl_init (); // 启动一个CURL会话
        curl_setopt ( $curl, CURLOPT_URL, $url ); // 要访问的地址
        curl_setopt ( $curl, CURLOPT_SSL_VERIFYPEER, FALSE ); // 对认证证书来源的检查
        curl_setopt ( $curl, CURLOPT_SSL_VERIFYHOST, FALSE ); // 从证书中检查SSL加密算法是否存在
        curl_setopt ( $curl, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)' ); // 模拟用户使用的浏览器
        curl_setopt($curl, CURLOPT_POST, true);
        curl_setopt ( $curl, CURLOPT_POSTFIELDS, http_build_query($params)); // Post提交的数据包
        curl_setopt ( $curl, CURLOPT_TIMEOUT, 30 ); // 设置超时限制防止死循环
        curl_setopt ( $curl, CURLOPT_RETURNTRANSFER, 1); // 获取的信息以文件流的形式返回
        $result = curl_exec ( $curl ); // 执行操作
        curl_close ( $curl ); // 关闭CURL会话

        $result = json_decode($result, true);
        return $result;
    }
}

Action

public function actionDemoMultiThread($condition){
     
        $debug = isset($condition['Debug']) && $condition['Debug'] ? true : false;
        $length = isset($condition['Length']) && $condition['Length'] ? $condition['Length'] : 1000;
        if($debug){
     
            //向redis队列中插入length个元素
        }

        $redis = Soulgame::$app->get('common_cache');
        echo "\n队列剩余长度：" . $redis->lSize('battleLogCallbackAsyncQueue');
        $st = millitime();
        $db_config = [
            'dsn' => 'mysql:host='.DB_HOST.';dbname='.DB_NAME,
            'username'  => DB_USER,
            'password'  => DB_PASSWORD,
            'charset'   => 'utf8',
            'collation' => 'utf8_general_ci',
            'prefix'    => '',
        ];

        $redis_config = [
            'host'=>REDIS_HOST,
            'port'=>REDIS_PORT,
            'passwd'=>REDIS_PASSWORD,
            'options'=>[\Redis::OPT_PREFIX=>REDIS_PREFIX, \Redis::OPT_SERIALIZER=>\Redis::SERIALIZER_PHP]
        ];


        $thread_group = [];
        $thread_group_num = 20;
        for($i = 0; $i < $thread_group_num; $i++){
     
            $thread_group[$i] = new DemoThread($db_config, $redis_config, $debug);
        }

        for($i = 0; $i < $thread_group_num; $i++) {
     
            $thread_group[$i]->start();
        }

        for($i = 0; $i < $thread_group_num; $i++) {
     
            $thread_group[$i]->join();
        }

        echo "\n执行时间：" . (millitime() - $st) . "ms 队列剩余长度：" . $redis->lSize('battleLogCallbackAsyncQueue') . "\n";
        exit;
    }

测试结果

root@qd004-backend-dev-php7:/data/game/lgy/h5game/caroline# /usr/local/php/bin/php /data/game/lgy/h5game/caroline/console/console.php job DemoMultiThread Debug:1
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking lrsphp7s1.soulgame.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx/1.6.2
Server Hostname:       xxxx
Server Port:            80

Document Path:          /lgy/h5game/caroline/api/web/index.php?c=battle&a=StartAloneLog&debug=1&test=1&UserId=11172325&Score=1&RoomType=3&StartTime=1522078511&EndTime=1522078516
Document Length:        187 bytes

Concurrency Level:      50
Time taken for tests:   0.948 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      470000 bytes
HTML transferred:       187000 bytes
Requests per second:    1054.46 [#/sec] (mean)
Time per request:       47.418 [ms] (mean)
Time per request:       0.948 [ms] (mean, across all concurrent requests)
Transfer rate:          483.98 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       2
Processing:     3   47  76.1     15     484
Waiting:        3   47  76.1     15     484
Total:          3   47  76.2     15     484

Percentage of the requests served within a certain time (ms)
  50%     15
  66%     30
  75%     45
  80%     62
  90%    132
  95%    209
  98%    335
  99%    392
 100%    484 (longest request)

队列剩余长度：1000
执行时间：369ms 队列剩余长度：0

开启20个线程，1000个Post请求花费时间369ms, 异步发送post请求数 接近 3000 op/s, 重要是测试阶段，4G 8G 配置的服务器 CPU负载不到1.0（swoole协程4.0跑满）。

---

1. https://blog.skl9.com/archives/323 ↩︎

2. https://blog.csdn.net/gavin_new/article/details/65444190 ↩︎