ansible
playbook
yaml
自动化运维
当下有许多的运维自动化工具( 配置管理 ),例如:Ansible、SaltStack、Puppet、Fabric 等。
Ansible 一种集成 IT 系统的配置管理、应用部署、执行特定任务的开源平台,是 AnsibleWorks 公司名下的项目,该公司由 Cobbler 及 Func 的作者于 2012 年创建成立。
Ansible 基于 Python 语言实现,由 Paramiko 和 PyYAML 两个关键模块构建。
Ansible 特点:
Ansible 与 SaltStack:
yum -y install ansible
ls /etc/ansible
ansible.cfg hosts roles
# ansible.cfg 是 Ansible 工具的配置文件;hosts 用来配置被管理的机器;roles 是一个目录,playbook 将使用它
ssh-keygen -t rsa
ssh-copy-id root@agent_host_ip
vim /etc/ansible/hosts
[Client]
angent_host_ip_1
angent_host_ip_2
shell > ansible Client -m ping # 操作 Client 组 ( all 为操作 hosts 文件中所有主机 ),-m 指定执行 ping 模块,下面是返回结果
192.168.12.129 | SUCCESS => {
"changed": false,
"ping": "pong"
}
# -i 指定 hosts 文件位置
# -u username 指定 SSH 连接的用户名
# -k 指定远程用户密码
# -f 指定并发数
# -s 如需要 root 权限执行时使用 ( 连接用户不是 root 时 )
# -K -s 时,-K 输入 root 密码
shell > vim /etc/ansible/hosts
www.abc.com # 定义域名
192.168.1.100 # 定义 IP
192.168.1.150:37268 # 指定端口号
[WebServer] # 定义分组
192.168.1.10
192.168.1.20
192.168.1.30
[DBServer] # 定义多个分组
192.168.1.50
192.168.1.60
Monitor ansible_ssh_port=12378 ansible_ssh_host=192.168.1.200 # 定义别名
# ansible_ssh_host 连接目标主机的地址
# ansible_ssh_port 连接目标主机的端口,默认 22 时无需指定
# ansible_ssh_user 连接目标主机默认用户
# ansible_ssh_pass 连接目标主机默认用户密码
# ansible_ssh_connection 目标主机连接类型,可以是 local 、ssh 或 paramiko
# ansible_ssh_private_key_file 连接目标主机的 ssh 私钥
# ansible_*_interpreter 指定采用非 Python 的其他脚本语言,如 Ruby 、Perl 或其他类似 ansible_python_interpreter 解释器
[webservers] # 主机名支持正则描述
www[01:50].example.com
[dbservers]
db-[a:f].example.com
shell > ansible-doc -l # 列出 Ansible 支持的模块
shell > ansible-doc ping # 查看该模块帮助信息
command 作为 Ansible 的默认模块,可以运行远程权限范围所有的 shell 命令,不支持管道符。
shell > ansible Client -m command -a "free -m" # 查看 Client 分组主机内存使用情况
script 的功能是在远程主机执行主控端存储的 shell 脚本文件,相当于 scp + shell 组合。
shell > ansible Client -m script -a "/home/test.sh 12 34" # 远程执行本地脚本
shell模块基本和command相同,但是shell支持管道符
shell > ansible Client -m shell -a "/home/test.sh" # 执行远程脚本
实现主控端向目标主机拷贝文件,类似于 scp 功能
shell > ansible Client -m copy -a "src=/home/test.sh dest=/tmp/ owner=root group=root mode=0755" # 向 Client 组中主机拷贝 test.sh 到 /tmp 下,属主、组为 root ,权限为 0755
获取远程文件状态信息,atime/ctime/mtime/md5/uid/gid 等信息
shell > ansible Client -m stat -a "path=/etc/syctl.conf"
实现在远程主机下载指定 URL 到本地,支持 sha256sum 文件校验
shell > ansible Client -m get_url -a "url=http://www.baidu.com dest=/tmp/index.html mode=0440 force=yes"
软件包管理
shell > ansible Client -m yum -a "name=curl state=latest"
远程主机 crontab 配置
shell > ansible Client -m cron -a "name='check dirs' hour='5,2' job='ls -alh > /dev/null'"
效果:
* 5,2 * * * ls -alh > /dev/null
远程主机分区挂载
shell > ansible Client -m mount -a "name=/mnt/data src=/dev/sd0 fstype=ext4 opts=ro state=present"
远程主机系统服务管理
shell > ansible Client -m service -a "name=nginx state=stoped"
shell > ansible Client -m service -a "name=nginx state=restarted"
shell > ansible Client -m service -a "name=nginx state=reloaded"
远程主机用户管理
shell > ansible Client -m user -a "name=wang comment='user wang'"
shell > ansible Client -m user -a "name=wang state=absent remove=yes" # 添加删除用户
name: tom
age: 21
gender: male
spourse:
name: lily
gender: female
children:
- name: susan
age: 2
gender: feamle
- name: sunny
age: 10
gender: male
vim /root/first.yml
- hosts: all
remote_user: root
vars: httpd_port=80
tasks:
- name: install httpd
yum: name=httpd state=present
- name: install php
yum: name=php state=present
- name: start httpd
service: name=httpd state=started enabled=true
hosts 定义单个主机或组,vars定义变量,remote_user定义执行命令的远程用户,tasks定义执行哪些命令,handlers定义调用哪些处理器
vars(变量):
变量命名: 字母数字下划线组成,只能以字母开头
变量种类:
facts(内置变量)
由远程主机发回的主机属性信息,这些信息被保存在ansible变量当中
例如:ansible 192.168.238.170 -m setup 来获取远程主机上的属性信息,这些属性信息保存在facts中
通过命令行传递
通过命令行传递:ansible-playbook test.yml –extra-vars “host=www user=tom“(如果剧本中已有此处定义的变量则会被覆盖)
通过roles传递
主机变量
在/etc/ansible/hosts中定义
[web1] 192.168.1.1 name=haha
组变量
[group_name:vars] foo=bar
hosts :
/etc/abible/hosts 中指定的远程主机,并用指定的属性进行连接
ansible_ssh_port 连接远程主机使用的端口 ansible_ssh_user 连接远程主机使用的用户 ansible_ssh_pass 连接远程主机使用的密码
cat /etc/ansible/hosts [web1] web1.hostname ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_pass=123 web2.hostname
vim /root/second.yml
- hosts: web1
remote_user: root
vars:
username: bob
password: 123
tasks:
- name: add user
user: name={{ username }} state=present
when: ansible_os_family == "Debian"
- name: set password
shell: echo {{ password }} |passwd --stdin {{ username }}
- name: install httpd php
yum: name={{ item }} state=present
with_items:
- httpd
- php
- name: add two users
user: name={{ item }} state=present groups={{ item.groups }}
with_items:
- { name: 'user1', groups: 'group1'}
- { name: 'user2', groups: 'group2'}
vim /root/third.yml
- hosts: web1
remote_user: root
vars:
httpd_port=80
tasks:
- name: install httpd
yum: name=httpd state=present
- name: install php
yum: name=php state=present
- name: copy config file
copy: src=/root/httpd.conf dest=/etc/httpd/conf/httpd.conf
notify: restart httpd
- name: start httpd
service: name=httpd state=started enabled=true
handlers:
- name: restart httpd
service: name=httpd state=restarted
上面的意思是copy中复制过去的文件跟远程主机上的文件不同,就通过notify调用handlers,即重启httpd服务。
handler是重启服务是最通用的用法
vim /etc/ansible/hosts
[web1]
192.168.1.1 http_port=80
vim /root/httpd.conf
……
Listen {{ http_port }}
……
vim /root/fourth.yml
- hosts: web1
remote_user: root
vars:
httpd_port=80
tasks:
- name: install httpd
yum: name=httpd state=present
- name: copy config file
template: src=/root/httpd.conf dest=/etc/httpd/conf/httpd.conf
notify: restart httpd
- name: start httpd
service: name=httpd state=started enabled=true
handlers:
- name: restart httpd
service: name=httpd state=restarted
templates:用于生成文本文件(配置文件)
模板文件中可使用jinja2表达式,表达式要定义在{{ }},也可以简单地仅执行变量替换
roles:roles用于实现“代码复用”,roles以特定的层次型格式组织起来的playbook元素(variables, tasks, templates,handlers);可被playbook以role的名字直接进行调用
roles的文件结构:
mkdir /root/roles
cd /root/roles
mkdir -p web1/{files, templayes, tasks, handlers, vars, meta}
vim web1/vars/main.yml
user: tom
group: tom
http_port: 8080
vim web1/tasks/main.yml
- name: install httpd
yum: name=httpd state=present
- name: copy config file
template: src=httpd.conf dest=/etc/httpd/conf/httpd.conf
notify: restart httpd
tags: conf
- name: start httpd
service: name=httpd state=started enabled=true
这里的template指的是相对路径-->web1/templates
tags可以在运行时指定标签任务
vim web1/handlers/main.yml
handlers:
- name: restart httpd
service: name=httpd state=restarted
vim web1/templates/httpd.conf
……
Listen {{ http_port }}
……
定义一个调用roles文件
vim /root/web1.yml
- hosts: web1
remote_user: root
roles:
- web1
- { role:web2, http_port:8080 }
hosts:web1 指在/etc/ansible/hosts中定义的组,上面有定义
roles: web1 指的是当前目录下的web1目录,也可通过role传递变量, 也可调用多个role
这样只需更改hosts的主机就可以实现不同主机的代码重用了
运行
ansible-playbook web1.yml
指定运行任务:
ansible-playbook -t conf web1.yml
shell > vim /etc/ansible/hosts
[mini]
129.139.153.78:16283
155.139.190.94:12573
shell > vim /etc/ansible/install_zabbix_agent.yml
---
- hosts: mini
roles:
- install_zabbix_agent
## 可以看到将要安装的主机组为 mini 组,角色为 install_zabbix_agent
shell > tree /etc/ansible/roles/install_zabbix_agent/
├── files
│ └── zabbix-2.4.5.tar.gz
├── tasks
│ └── main.yml
├── templates
│ ├── zabbix_agentd
│ └── zabbix_agentd.conf
└── vars
└── main.yml
## 建立 files 目录,存放编译安装过的 zabbix_agent 目录的压缩文件,用于拷贝到远程主机
## 建立 tasks 目录,用于编写将要执行的任务
## 建立 templates 目录,用于存放可变的模板文件
## 建立 vars 目录,用于存放变量信息
shell > cat /etc/ansible/roles/install_zabbix_agent/tasks/main.yml
---
- name: Install Software
yum: name={{ item }} state=latest
with_items:
- libcurl-devel
- name: Create Zabbix User
user: name={{ zabbix_user }} state=present createhome=no shell=/sbin/nologin
- name: Copy Zabbix.tar.gz
copy: src=zabbix-{{ zabbix_version }}.tar.gz dest={{ zabbix_dir }}/src/zabbix-{{ zabbix_version }}.tar.gz owner=root group=root
- name: Uncompression Zabbix.tar.gz
shell: tar zxf {{ zabbix_dir }}/src/zabbix-{{ zabbix_version }}.tar.gz -C {{ zabbix_dir }}/
- name: Copy Zabbix Start Script
template: src=zabbix_agentd dest=/etc/init.d/zabbix_agentd owner=root group=root mode=0755
- name: Copy Zabbix Config File
template: src=zabbix_agentd.conf dest={{ zabbix_dir }}/zabbix/etc/zabbix_agentd.conf owner={{ zabbix_user }} group={{ zabbix_user }} mode=0644
- name: Modify Zabbix Dir Permisson
file: path={{ zabbix_dir }}/zabbix owner={{ zabbix_user }} group={{ zabbix_user }} mode=0755 recurse=yes
- name: Start Zabbix Service
shell: /etc/init.d/zabbix_agentd start
- name: Add Boot Start Zabbix Service
shell: chkconfig --level 35 zabbix_agentd on
shell > cat /etc/ansible/roles/install_zabbix_agent/vars/main.yml
zabbix_dir: /usr/local
zabbix_version: 2.4.5
zabbix_user: zabbix
zabbix_port: 10050
zabbix_server_ip: 131.142.101.120
shell > cat /etc/ansible/roles/install_zabbix_agent/templates/zabbix_agentd
#!/bin/bash
#
# chkconfig: - 90 10
# description: Starts and stops Zabbix Agent using chkconfig
# Tested on Fedora Core 2 - 5
# Should work on all Fedora Core versions
#
# @name: zabbix_agentd
# @author: Alexander Hagenah
# @created: 18.04.2006
#
# Modified for Zabbix 2.0.0
# May 2012, Zabbix SIA
#
# Source function library.
. /etc/init.d/functions
# Variables
# Edit these to match your system settings
# Zabbix-Directory
BASEDIR={{ zabbix_dir }}/zabbix
# Binary File
BINARY_NAME=zabbix_agentd
# Full Binary File Call
FULLPATH=$BASEDIR/sbin/$BINARY_NAME
# PID file
PIDFILE=/tmp/$BINARY_NAME.pid
# Establish args
ERROR=0
STOPPING=0
#
# No need to edit the things below
#
# application checking status
if [ -f $PIDFILE ] && [ -s $PIDFILE ]
then
PID=`cat $PIDFILE`
if [ "x$PID" != "x" ] && kill -0 $PID 2>/dev/null && [ $BINARY_NAME == `ps -e | grep $PID | awk '{print $4}'` ]
then
STATUS="$BINARY_NAME (pid `pidof $APP`) running.."
RUNNING=1
else
rm -f $PIDFILE
STATUS="$BINARY_NAME (pid file existed ($PID) and now removed) not running.."
RUNNING=0
fi
else
if [ `ps -e | grep $BINARY_NAME | head -1 | awk '{ print $1 }'` ]
then
STATUS="$BINARY_NAME (pid `pidof $APP`, but no pid file) running.."
else
STATUS="$BINARY_NAME (no pid file) not running"
fi
RUNNING=0
fi
# functions
start() {
if [ $RUNNING -eq 1 ]
then
echo "$0 $ARG: $BINARY_NAME (pid $PID) already running"
else
action $"Starting $BINARY_NAME: " $FULLPATH
touch /var/lock/subsys/$BINARY_NAME
fi
}
stop() {
echo -n $"Shutting down $BINARY_NAME: "
killproc $BINARY_NAME
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$BINARY_NAME
RUNNING=0
}
# logic
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status $BINARY_NAME
;;
restart)
stop
sleep 10
start
;;
help|*)
echo $"Usage: $0 {start|stop|status|restart|help}"
cat <$BINARY_NAME
stop - stop $BINARY_NAME
status - show current status of $BINARY_NAME
restart - restart $BINARY_NAME if running by sending a SIGHUP or start if not running
help - this screen
EOF
exit 1
;;
esac
exit 0
shell > cat /etc/ansible/roles/install_zabbix_agent/templates/zabbix_agentd.conf
# This is a config file for the Zabbix agent daemon (Unix)
# To get more information about Zabbix, visit http://www.zabbix.com
############ GENERAL PARAMETERS #################
### Option: PidFile
# Name of PID file.
#
# Mandatory: no
# Default:
# PidFile=/tmp/zabbix_agentd.pid
### Option: LogFile
# Name of log file.
# If not set, syslog is used.
#
# Mandatory: no
# Default:
# LogFile=
LogFile=/tmp/zabbix_agentd.log
### Option: LogFileSize
# Maximum size of log file in MB.
# 0 - disable automatic log rotation.
#
# Mandatory: no
# Range: 0-1024
# Default:
# LogFileSize=1
### Option: DebugLevel
# Specifies debug level
# 0 - basic information about starting and stopping of Zabbix processes
# 1 - critical information
# 2 - error information
# 3 - warnings
# 4 - for debugging (produces lots of information)
#
# Mandatory: no
# Range: 0-4
# Default:
# DebugLevel=3
### Option: SourceIP
# Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=
### Option: EnableRemoteCommands
# Whether remote commands from Zabbix server are allowed.
# 0 - not allowed
# 1 - allowed
#
# Mandatory: no
# Default:
# EnableRemoteCommands=0
### Option: LogRemoteCommands
# Enable logging of executed shell commands as warnings.
# 0 - disabled
# 1 - enabled
#
# Mandatory: no
# Default:
# LogRemoteCommands=0
##### Passive checks related
### Option: Server
# List of comma delimited IP addresses (or hostnames) of Zabbix servers.
# Incoming connections will be accepted only from the hosts listed here.
# If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
#
# Mandatory: no
# Default:
# Server=
Server={{ zabbix_server_ip }}
### Option: ListenPort
# Agent will listen on this port for connections from the server.
#
# Mandatory: no
# Range: 1024-32767
# Default:
# ListenPort=10050
ListenPort={{ zabbix_port }}
### Option: ListenIP
# List of comma delimited IP addresses that the agent should listen on.
# First IP address is sent to Zabbix server if connecting to it to retrieve list of active checks.
#
# Mandatory: no
# Default:
# ListenIP=0.0.0.0
### Option: StartAgents
# Number of pre-forked instances of zabbix_agentd that process passive checks.
# If set to 0, disables passive checks and the agent will not listen on any TCP port.
#
# Mandatory: no
# Range: 0-100
# Default:
# StartAgents=3
##### Active checks related
### Option: ServerActive
# List of comma delimited IP:port (or hostname:port) pairs of Zabbix servers for active checks.
# If port is not specified, default port is used.
# IPv6 addresses must be enclosed in square brackets if port for that host is specified.
# If port is not specified, square brackets for IPv6 addresses are optional.
# If this parameter is not specified, active checks are disabled.
# Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=
#ServerActive=127.0.0.1:10051
### Option: Hostname
# Unique, case sensitive hostname.
# Required for active checks and must match hostname as configured on the server.
# Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=
Hostname={{ ansible_all_ipv4_addresses[1] }}
### Option: HostnameItem
# Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.
# Does not support UserParameters or aliases.
#
# Mandatory: no
# Default:
# HostnameItem=system.hostname
### Option: HostMetadata
# Optional parameter that defines host metadata.
# Host metadata is used at host auto-registration process.
# An agent will issue an error and not start if the value is over limit of 255 characters.
# If not defined, value will be acquired from HostMetadataItem.
#
# Mandatory: no
# Range: 0-255 characters
# Default:
# HostMetadata=
### Option: HostMetadataItem
# Optional parameter that defines an item used for getting host metadata.
# Host metadata is used at host auto-registration process.
# During an auto-registration request an agent will log a warning message if
# the value returned by specified item is over limit of 255 characters.
# This option is only used when HostMetadata is not defined.
#
# Mandatory: no
# Default:
# HostMetadataItem=
### Option: RefreshActiveChecks
# How often list of active checks is refreshed, in seconds.
#
# Mandatory: no
# Range: 60-3600
# Default:
# RefreshActiveChecks=120
### Option: BufferSend
# Do not keep data longer than N seconds in buffer.
#
# Mandatory: no
# Range: 1-3600
# Default:
# BufferSend=5
### Option: BufferSize
# Maximum number of values in a memory buffer. The agent will send
# all collected data to Zabbix Server or Proxy if the buffer is full.
#
# Mandatory: no
# Range: 2-65535
# Default:
# BufferSize=100
### Option: MaxLinesPerSecond
# Maximum number of new lines the agent will send per second to Zabbix Server
# or Proxy processing 'log' and 'logrt' active checks.
# The provided value will be overridden by the parameter 'maxlines',
# provided in 'log' or 'logrt' item keys.
#
# Mandatory: no
# Range: 1-1000
# Default:
# MaxLinesPerSecond=100
############ ADVANCED PARAMETERS #################
### Option: Alias
# Sets an alias for an item key. It can be used to substitute long and complex item key with a smaller and simpler one.
# Multiple Alias parameters may be present. Multiple parameters with the same Alias key are not allowed.
# Different Alias keys may reference the same item key.
# For example, to retrieve the ID of user 'zabbix':
# Alias=zabbix.userid:vfs.file.regexp[/etc/passwd,^zabbix:.:([0-9]+),,,,\1]
# Now shorthand key zabbix.userid may be used to retrieve data.
# Aliases can be used in HostMetadataItem but not in HostnameItem parameters.
#
# Mandatory: no
# Range:
# Default:
### Option: Timeout
# Spend no more than Timeout seconds on processing
#
# Mandatory: no
# Range: 1-30
# Default:
Timeout=20
### Option: AllowRoot
# Allow the agent to run as 'root'. If disabled and the agent is started by 'root', the agent
# will try to switch to the user specified by the User configuration option instead.
# Has no effect if started under a regular user.
# 0 - do not allow
# 1 - allow
#
# Mandatory: no
# Default:
# AllowRoot=0
### Option: User
# Drop privileges to a specific, existing user on the system.
# Only has effect if run as 'root' and AllowRoot is disabled.
#
# Mandatory: no
# Default:
# User=zabbix
### Option: Include
# You may include individual files or all files in a directory in the configuration file.
# Installing Zabbix will create include directory in /usr/local/etc, unless modified during the compile time.
#
# Mandatory: no
# Default:
# Include=
# Include=/usr/local/etc/zabbix_agentd.userparams.conf
# Include=/usr/local/etc/zabbix_agentd.conf.d/
# Include=/usr/local/etc/zabbix_agentd.conf.d/*.conf
####### USER-DEFINED MONITORED PARAMETERS #######
### Option: UnsafeUserParameters
# Allow all characters to be passed in arguments to user-defined parameters.
# 0 - do not allow
# 1 - allow
#
# Mandatory: no
# Range: 0-1
# Default:
UnsafeUserParameters=1
### Option: UserParameter
# User-defined parameter to monitor. There can be several user-defined parameters.
# Format: UserParameter=,
# See 'zabbix_agentd' directory for examples.
#
# Mandatory: no
# Default:
# UserParameter=
####### LOADABLE MODULES #######
### Option: LoadModulePath
# Full path to location of agent modules.
# Default depends on compilation options.
#
# Mandatory: no
# Default:
# LoadModulePath=${libdir}/modules
### Option: LoadModule
# Module to load at agent startup. Modules are used to extend functionality of the agent.
# Format: LoadModule=
# The modules must be located in directory specified by LoadModulePath.
# It is allowed to include multiple LoadModule parameters.
#
# Mandatory: no
# Default:
# LoadModule=
shell > ansible-playbook /etc/ansible/install_zabbix_agent.yml
PLAY [mini] *******************************************************************
GATHERING FACTS ***************************************************************
ok: [129.139.153.78]
ok: [155.139.190.94]
TASK: [install_zabbix_agent | Install Software] *******************************
changed: [155.139.190.94] => (item=libcurl-devel)
changed: [129.139.153.78] => (item=libcurl-devel)
TASK: [install_zabbix_agent | Create Zabbix User] *****************************
changed: [129.139.153.78]
changed: [155.139.190.94]
TASK: [install_zabbix_agent | Copy Zabbix.tar.gz] *****************************
changed: [129.139.153.78]
changed: [155.139.190.94]
TASK: [install_zabbix_agent | Uncompression Zabbix.tar.gz] ********************
changed: [129.139.153.78]
changed: [155.139.190.94]
TASK: [install_zabbix_agent | Copy Zabbix Start Script] ***********************
changed: [155.139.190.94]
changed: [129.139.153.78]
TASK: [install_zabbix_agent | Copy Zabbix Config File] ************************
changed: [129.139.153.78]
changed: [155.139.190.94]
TASK: [install_zabbix_agent | Modify Zabbix Dir Permisson] ********************
changed: [155.139.190.94]
changed: [129.139.153.78]
TASK: [install_zabbix_agent | Start Zabbix Service] ***************************
changed: [129.139.153.78]
changed: [155.139.190.94]
TASK: [install_zabbix_agent | Add Boot Start Zabbix Service] ******************
changed: [129.139.153.78]
changed: [155.139.190.94]
PLAY RECAP ********************************************************************
155.139.190.94 : ok=10 changed=9 unreachable=0 failed=0
129.139.153.78 : ok=10 changed=9 unreachable=0 failed=0
## 关注一下,启动脚本跟配置文件中变量的引用。
## 完成安装,可以去客户机检查效果了 !
这是本人的个人博客,欢迎关注:
CSDN:弓昭的技术博客
简书:弓昭的简书
掘金:弓昭的个人主页
如果有问题可以Email或者留言与我交流
[email protected]
PS:每篇博文都有PDF版本,想要的朋友可以email我
参考链接:
小怪兽的技术博客