压力测试前需要对大数据环境、日志和存储数据库进行清理、检查环境是否可用、保存测试前的环境状态。
压力测试过程中需要定期检查PARQUET文件生成情况、报表生成情况、HBASE数据生成情况、是否有SPARK APP FAILED、收集错误日志等。
压力测试后需要检查PARQUET文件生成情况、报表生成情况、HBASE数据生成情况、是否有SPARK APP FAILED、收集错误日志、查看APP执行时间间隔、各主机CPU内存网络磁盘状况等。
基础脚本，eea_2_all.sh，分发命令到HDP集群各节点主机执行，内容如下，注意基础脚本运行于HDP第一台主机（与其他主机做免密处理）。
#!/bin/bash
#数组中用主机名亦可
HOSTS=(192.168.1.4 192.168.1.5 192.168.1.6 192.168.1.8 192.168.1.9 192.168.1.10 192.168.1.11 192.168.1.12 192.168.1.13)
COMMAND=$1
PROMPT=$2
if [ $# == 0 ] ; then
COMMAND="/bin/rm -rf /var/log/rmslog/*;/bin/echo '' > /var/log/rmslog/debug.log;/bin/chmod 666 /var/log/rmslog/debug.log"
PROMPT="Clear debug log"
fi
count=1
for host in ${HOSTS[@]}
do
echo "-----------------------eea-younge-${count}.test--$host: ${PROMPT}----------------------"
ssh $host ${COMMAND}
let count++
done
1.测试前的准备工作包括，
a.确保Hmaster上面的Hbase Thrift Server处于运行状态，命令如下。
. ssh 192.168.1.6 "if test $(ps -ef | grep 'hbase.thrift.ThriftServer start' | wc -l ) -eq 2 then ; echo 'Hbase Thrift Server is running.' else nohup hbase thrift start & fi"
b.对大数据环境、本地日志和存储数据库进行清理，检查YARN中应用情况。
#清理Hbase表
#其中cmd.hbase存放的是按行存储的hbase shell命令，最后一行需为exit，用于退出hbase shell。
nohup hbase shell hbase/cmds.hbase > result/clear_hbase_${now}.log
#清理Hdfs Parquet文件，shell/clear_parquets.sh
#!/bin/bash
DeleteHDFSFolder()
{
sudo -u spark hadoop fs -rm -r -f $1
if [ $? -ne 0 ] ; then
LogMessage "Can not create Delete files and folders $1."
fi
}
CreateHDFSFolder()
{
sudo -u spark hadoop fs -mkdir -p $1
if [ $? -ne 0 ] ; then
LogMessage "Can not create HDFS folders $1."
fi
}
DeleteHDFSFolder "/user/spark/eea/alarm/*"
DeleteHDFSFolder "/user/spark/eea/fine/*"
DeleteHDFSFolder "/user/spark/eea/trip/*"
DeleteHDFSFolder "/user/spark/eea/event/*"
DeleteHDFSFolder "/user/spark/eea/collector/*"
DeleteHDFSFolder "/user/spark/eea/rawdata/*"
sudo -u hdfs hdfs dfs -expunge
CreateHDFSFolder "/user/spark/eea/alarm/enrichment/"
CreateHDFSFolder "/user/spark/eea/fine/enrichment/"
CreateHDFSFolder "/user/spark/eea/trip/enrichment/"
#清理各主机的本地日志
. ${shell_path}/eea_2_all.sh
#清理Redis数据库，其中cmds.redis中存放的为按行存储命令，本篇中仅一条命令flushall
nohup /bin/cat redis/cmds.redis | redis-cli -h 192.168.1.5 -a runoob > result/clear_redis_${now}.log
#清理PostgreSQL工作日志表
export PGPASSWORD=pass123456
nohup psql -h 192.168.1.155 -p 5432 -U test -d test -f sql/query_truncate_app_orchestration.sql > result/test_clear_work_log_${now}.log
#query_truncate_work_log.sql 内容如下，
select * from app_orchestration;
truncate app_orchestration;
#检查磁盘挂载情况
nohup shell/eea_2_all.sh "mount | grep \"data/eea\"" "Mount Directory /data/eea Check" > result/mount_check_${now}.txt
#检查各主机磁盘空间情况
nohup shell/eea_2_all.sh "df -hT" "Disk Usage" 2>&1 > result/"vms_disk_usage_${now}.txt"
#检查数据库空间情况
psql -h 192.168.1.155 -p 5432 -U test -d test -f sql/check_databases_size.sql > result/rms_eea_databases_size_${now}.txt
#check_databases_size.sql 内容如下，
SELECT d.datname AS Name, pg_catalog.pg_get_userbyid(d.datdba) AS Owner,
CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')
THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))
ELSE 'No Access' END AS SIZE
FROM pg_catalog.pg_database d WHERE d.datname in ('test')
ORDER BY
CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')
THEN pg_catalog.pg_database_size(d.datname)
ELSE NULL
END DESC -- nulls first
LIMIT 20 ;
#检查数据库表空间情况
psql -h 192.168.1.155 -p 5432 -U test -d test -f sql/check_tables_size.sql > result/rms_eea_tables_size_${now}.txt
#check_tables_size.sql内容如下，
SELECT
table_schema || '.' || table_name AS table_full_name,
pg_size_pretty(pg_total_relation_size('"' || table_schema || '"."' || table_name || '"')) AS size
FROM information_schema.tables WHERE table_schema='public'
ORDER BY
pg_total_relation_size('"' || table_schema || '"."' || table_name || '"') DESC;
#检查YARN中失败的应用列表
nohup yarn application -list -appStates FAILED > result/yarn_app_failed_list.log
#检查YARN中KILLED的应用列表
nohup yarn application -list -appStates KILLED > result/yarn_app_killed_list.log
压测过程中检查见后续文章。。。
