程序员瑞士军刀之 Fabric

Fabric是一个Python库, 也是一个命令行工具, 通过 SSH 来做应用程序的部署和系统管理任务

它可以执行本地的远程的系统命令, 上传下载文件, 以及其他能用Python编程完成的任务

其实它是一个工具框架, 启动时会默认执行一个 python 文件 fabfile.py

安装

先安装好 python3.x， 再安装 fabric3

pip install fabric3

快速上手

简单写个小例子

$vi fabfile

    from fabric.api import *

    env.hosts = ['10.224.64.106']
    env.user   = "root"
    env.password = "pass"
    
    def freedisk(param='-h'):
        cmd = 'df ' + param
        run(cmd)
    
    def listfile(folder='~'):
        cmd = 'ls -l ' + folder
        run(cmd)
        
    def pullcodes(folder='/workspace/cpp/snippets'):
        with cd(folder):
            run("git pull origin master")

# 察看远程服务器上的磁盘剩余空间
$ fab listfile:folder=/home/walter    

• 为安全起见, 不用在文件中存放密码, 在命令行提示输入

  $ fab -u root -I -H 10.224.64.106 freedisk

• 更好的做法是把本机私钥预先拷贝到目标服务器上, 这样就不用输入密码了

    1. 在本机上生成公钥  ~/.ssh/id_rsa.pub
    ssh-keygen -t rsa
    
    2. 拷贝此公钥到目标服务器 10.224.64.106 上
    scp id_rs.pub [email protected]:/root
    
    3. 目标服务器 10.224.64.106 上
    cat id_rsa.pub >> ~/.ssh/authorized_keys
    chmod 700 ~/.ssh/authorized_keys

常用方法

• run (fabric.operations.run)
• sudo (fabric.operations.sudo)
• local (fabric.operations.local)
• get (fabric.operations.get)
• put (fabric.operations.put)
• prompt (fabric.operations.prompt)
• reboot (fabric.operations.reboot)

常用函数

• cd (fabric.context_managers.cd)
• lcd (fabric.context_managers.lcd)
• path (fabric.context_managers.path)
• settings (fabric.context_managers.settings)
• prefix (fabric.context_managers.prefix)

高阶用法

设置角色role来指定远程的服务器范围
或者直接用字典由输入参数指定, 例如:

# usage:  
# fab localpull:rtc
# fab checkfiles:hf2
from fabric.api import *
from fabric.context_managers import *
from fabric.contrib.console import confirm 


env.user = 'root'
env.roledefs = {
    'qa': ['[email protected]:22'],
    'dev': ['[email protected]:22']
}

env.passwords = { 
    '[email protected]:22': 'pass',
    '[email protected]:22': 'pass',
    '[email protected]:22': 'pass'
  } 

@roles('dev')
@task
def localpull(app='web'):
    if app == 'web':
        code_dir = '/workspace/walter/hfweb'
        with lcd(code_dir):
            local("git pull origin master")
    elif app == 'rtc':
        code_dir = '/workspace/walter/hfrtc'
        with lcd(code_dir):
            local("git pull origin master")
            local("git branch -l")

test_servers = {'hf1':['[email protected]:22'],
    'hf2':['[email protected]:22'],
    'hf3':['[email protected]:22']}

@task
def listfiles():
    run("ls -l")

@task
def checkfiles(target_env='hf2'):
    execute("listfiles", hosts=test_servers[target_env])

示例

例1:批量上传下载文件

from fabric.api import *
from fabric.context_managers import *
from fabric.contrib.console import confirm 

env.user='root'
env.hosts=['10.224.64.106'] 
env.passwords = { 
    '[email protected]:22': 'password'
  } 

local_dir='/workspace/cpp/codelab'
remote_dir = '/opt/cpp/codelab'
file_list = [
    'src/FileUtils.cpp',
    'src/FileUtils.h',
    'src/Makefile.am',
    'src/StringUtils.cpp'
]

@task
def hostinfo():
    run('uname -s')
          
@task
def upload(): #upload file task 
    with cd(remote_dir) :
        for filename in file_list:
            local_file  = local_dir  + "/" + filename
            remote_file = remote_dir + "/" + filename
            #print local_file, " to ", remote_file
            with settings(warn_only=True):    #when upload error,continue 
                result = put(local_file, remote_file) 
            if result.failed and not confirm("put file failed,Continue[Y/N]?"): 
                abort("Aborting file put task!")


@task
def download(): #upload file task 
    with cd(remote_dir) :
        for filename in file_list:
            local_file  = local_dir  + "/" + filename
            remote_file = remote_dir + "/" + filename
            #print local_file, " to ", remote_file
            with settings(warn_only=True):    #when upload error,continue 
                result = get(remote_file,local_file)
            if result.failed and not confirm("put file failed,Continue[Y/N]?"): 
                abort("Aborting file put task!")

例2: 创建 gitbook 目录, 以使用 markdown 来写作

from fabric.api import *
#import SimpleHTTPServer
import http.server
import socketserver
import os
from utils import chapters

BOOK_CONTENT_FOLDER = './book/content'
BOOK_OUTPUT_FOLDER = './book/output'

@task
def build_book(is_open_index=False):
    local( "rm -rf %s" % BOOK_OUTPUT_FOLDER)
    local( "mkdir -p %s" % BOOK_OUTPUT_FOLDER)

    cmd = "gitbook build %s %s --log=debug --debug" % (BOOK_CONTENT_FOLDER, BOOK_OUTPUT_FOLDER)
    local(cmd)
    if is_open_index:
        local("open %s/index.html" % BOOK_OUTPUT_FOLDER)

@task
def build_pdf():
    cmd = "gitbook pdf ./book/content/"
    local(cmd)


@task
def build_epub():
    cmd = "gitbook epub ./book/content/"
    local(cmd)
    local("ebook-convert book.epub book.docx")

@task
def init_book():
    create_templates()


@task
def serve_book(port=8000):

    web_dir = os.path.join(os.path.dirname(__file__), BOOK_OUTPUT_FOLDER)
    os.chdir(web_dir)

    Handler = http.server.SimpleHTTPRequestHandler
    httpd = socketserver.TCPServer(("", port), Handler)
    print("serving at port", port)
    httpd.serve_forever()

def md2rst(mdfile):
    rstfile = mdfile[:-3];
    cmd = "pandoc --from=markdown --to=rst --output=%s.md %s" % (rstfile, mdfile)
    print(cmd)
    local(cmd)

def rst2md(rstfile):
    mdfile = rstfile[:-3];
    cmd = "pandoc --from=rst --to=markdown --output=%s.md %s" % (mdfile, rstfile)
    print(cmd)
    local(cmd)

def create_templates():
    create_chapter(chapters.folder1, chapters.files1)
    create_chapter(chapters.folder2, chapters.files2)
    create_chapter(chapters.folder3, chapters.files3)
    create_chapter(chapters.folder4, chapters.files4)
    create_chapter(chapters.folder5, chapters.files5)

def create_chapter(folder, files):
    print("--- create chapter in %s ---" % folder)
 
    cmd = "mkdir -p %s/%s" % (BOOK_CONTENT_FOLDER, folder)
    local(cmd)

    for file in files:
        cmd = "touch %s/%s/%s" % (BOOK_CONTENT_FOLDER, folder, file)
        local(cmd)

例3. 用 Fabric 玩转 docker 基本命令

from fabric.api import *
from fabric.context_managers import *
from fabric.contrib.console import confirm
import os, subprocess

local_path = os.path.dirname(os.path.abspath(__file__))
local_dir = os.getcwd()

backend_service_ports={
"tomcat": "8080",
"kanban": "8080",
"cassandra": "9042",
"elasticsearch": "9200 9300",
"influxdb": "8086",
"postgres": "5432",
"rabbitmq": "4369 5671 5672 15671 15672 25672",
"redis": "6379",
"riak": "8087 8098",
"kafka-zookeeper": "2181 9092"
}

need_print_cmd=True
only_display_cmd=False

docker_image_prefix="walterfan-"
docker_container_prefix="msa-"

restart_policy="--restart always"
jenkins_volume_mapping = "/o/jenkins:/var/jenkins_home"
jenkins_container_name="jenkins"
jenkins_image_name="walterfan-jenkins"

def run_cmd(cmd):
    if(need_print_cmd):
        print(cmd)
    if not only_display_cmd:
        local(cmd)


@task
def jenkins_build():
    docker_build("jenkins")

@task
def jenkins_run(listen_port="1980"):
    cmd = "docker run %s -v %s -p %s:8080 -p 50000:50000 --name=%s -d %s" % (restart_policy, jenkins_volume_mapping, listen_port, jenkins_container_name, jenkins_image_name)
    run_cmd(cmd)


@task
def jenkins_start():
    cmd = "docker start %s" % jenkins_container_name
    run_cmd(cmd)

@task
def jenkins_stop():
    cmd = "docker stop %s" % jenkins_container_name
    local(cmd)
    #cmd = "docker cp jenkins-container:/var/log/jenkins/jenkins.log jenkins.log"
    #local(cmd)

@task
def jenkins_remove():
    docker_remove(jenkins_container_name)

@task
def jenkins_commit(message):
    cmd = "docker commit -m \"%s\" %s walterfan/jenkins:1.0" % (message, jenkins_container_name)

@task
def jenkins_check():
    cmd = "docker exec %s ps -ef | grep java" % jenkins_container_name
    print(cmd)
    local(cmd)

    cmd = "docker exec %s cat /var/jenkins_home/secrets/initialAdminPassword" % jenkins_container_name
    print(cmd)
    local(cmd)


#-----------------------------grafana influx --------------------------#
@task
def graflux_build():
    cmd = "docker build --tag %s docker/%s" % ("graflux", "graflux")
    run_cmd(cmd)


@task
def graflux_start():
    grafana_port = 3000
    influx_api_port = 8086
    influx_web_port = 8083
    cmd = "docker run --name local-graflux -d -p %d:3000 -p %d:8086 -p %d:8083 graflux" % (grafana_port, influx_api_port, influx_web_port)
    print(cmd)
    local(cmd)

@task
def influx():
    """
    execute the influx command in graflux docker
    """
    cmd = "docker exec -it local-graflux influx"
    run_cmd(cmd)

@task
def graflux_bash():
    """
    execute the /bin/bash in graflux docker
    """
    cmd = "docker exec -it local-graflux /bin/bash"
    run_cmd(cmd)

@task
def graflux_stop():
    #cmd = "docker stop local-graflux"
    docker_remove("local-graflux")

@task
def redis_cli():
    cmd = "docker exec -it local-redis redis-cli"
    local(cmd)
@task
def redis_bash():
    cmd = "docker exec -it local-redis /bin/bash"
    local(cmd)


@task
def cassandra_cql(cql=''):
    cmd = "docker exec -it local-cassandra /usr/bin/cqlsh "
    if cql:
        cmd = cmd + " -e '%s'" % cql
    local(cmd)

@task
def mysql_cli(usr='root'):
    cmd = "docker exec -it local-mysql /usr/bin/mysql -u %s -p" % usr
    local(cmd)

@task
def mysql_bash():
    cmd = "docker exec -it local-mysql /bin/bash"
    local(cmd)
#---------------------------- freeswitch -------------------------------#
#bettervoice/freeswitch-container   1.6.16
@task
def freeswitch_start():
    cmd = "sudo docker run --name freeswitch -p 5060:5060/tcp -p 5060:5060/udp -p 5080:5080/tcp -p 5080:5080/udp -p 8021:8021/tcp \
    -p 7443:7443/tcp -p 60535-65535:60535-65535/udp \
    -v %s/etc/freeswitch:/usr/local/freeswitch/conf bettervoice/freeswitch-container:1.6.16" % local_path
    print(cmd)
    local(cmd)

@task
def freeswitch_stop():
    docker_remove(freeswitch)
#-----------------------------------------------------------#
@task
def start_services():
    cmd = "docker-compose up -d"
    run_cmd(cmd)

@task
def stop_services():
    cmd = "docker-compose down -v"
    run_cmd(cmd)
#----------------------------- general command ----------------

@task
def link_war(war_package, war_name):
    cmd = "docker exec tomcat ln -s %s/%s /usr/local/tomcat/webapps/%s" % (local_path, war_package, war_name)
    local(cmd)

@task
def deploy_war(war_package, war_name):
    cmd = "docker cp %s/%s tomat:/usr/local/tomcat/webapps/%s" % (local_path, war_package, war_name)
    local(cmd)

@task
def undeploy_war(war_name):
    cmd = "docker exec tomcat rm -rf /usr/local/tomcat/webapps/%s" % (war_name)
    local(cmd)
    cmd = "docker exec tomcat rm -f /usr/local/tomcat/webapps/%s.war" % (war_name)
    local(cmd)
#----------------------------- general commands ---
def get_container_id(container_name):
    str_filter = "-aqf name=%s" % container_name;
    arr_cmd = ["docker", "ps", str_filter]
    container_id = subprocess.check_output(arr_cmd).strip()
    return container_id

def get_port_args(service_name="kanban", increment=0):
    str_port = ""
    ports = backend_service_ports[service_name]
    if ports:
        arr_port = ports.split("\\s")
        for port in arr_port:
            str_port = str_port + "-p %s:%d" %(port, int(port) + int(increment))
    return str_port


@task
def docker_rename(old_name, new_name):
    cmd = "docker tag %s %s" % (old_name, new_name)
    run_cmd(cmd)


@task
def docker_build(service_name="local-tomcat"):
    docker_image_name = docker_image_prefix + service_name
    cmd = "docker build --tag %s docker/%s" % (docker_image_name, service_name)
    run_cmd(cmd)


@task
def docker_run(service_name="local-tomcat", volume_args="-v /workspace:/workspace"):
    port_args = get_port_args(service_name)

    docker_container_name = docker_container_prefix + service_name
    docker_image_name = docker_image_prefix + service_name

    cmd = "docker run %s %s %s -d --name %s %s" % (restart_policy, volume_args, port_args, docker_container_name, docker_image_name)
    run_cmd(cmd)

@task
def docker_stop(container_name="local-tomcat"):
    cmd = "docker stop %s" % (container_name)
    run_cmd(cmd)

@task
def docker_list():
    cmd = "docker ps"
    run_cmd(cmd)

@task
def docker_exec(container_name="local-tomcat", instruction="/bin/bash"):

    instruction = "/bin/bash"
    cmd = "docker exec -it %s %s" % (container_name,    instruction)
    run_cmd(cmd)

@task
def docker_remove(container_name="kanban"):
    cmd1 = "docker kill %s|| true" % container_name
    run_cmd(cmd1)

    cmd2 = "docker rm -v %s || true" % container_name
    run_cmd(cmd2)

@task
def docker_commit(container_id, image_name, message=""):
    cmd = "docker commit -m \"%s\" %s %s" % (message, container_id, image_name)
    run_cmd(cmd)

@task
def docker_install():
    #cmd  ="brew remove docker && brew upgrade"
    cmd = "brew cask install docker && open /Applications/Docker.app"
    run_cmd(cmd)

@task
def help():
    print("examples:\tfab docker_run:cassandra,\"-v /opt:/workspace\" ")

FAQ

问题1: 切换环境

写一个字典对象,在参数里传入环境类型, 再组成所需的环境变量

environments = {
    "integration": {
         "ApiServiceUrl" :"https://checklist-int.example.com/checklist/api/v1", 
         "environment":"integration"
    },
    "production":{
         "ApiServiceUrl" :"https://checklist.example.com/checklist/api/v1", 
         "environment":"production"
    },
    "lab":{
         "ApiServiceUrl" :"https://checklist-lab.example.com/checklist/api/v1", 
         "environment":"lab"
    },
    "local":{
         "ApiServiceUrl" :"https://localhost:2008/checklist/api/v1", 
         "environment":"lab"
    }

}

def get_env_vars(env_type):
    defined_vars = " "
    for key, value in environments[env_type].iteritems():
        defined_vars = defined_vars + " -D%s=%s" %(key, value)
    return defined_vars;

问题2: 如何读取配置文件

• 以 json file 为例

class ProvisionConfig:
    def __init__(self, json_file):

        self.read_config(json_file)

        self.base_path = self.config_data['basePath']
        self.username = self.config_data['username']
        self.locale = self.config_data['locale']

    def read_config(self, json_file):
        json_data=open(json_file)
        self.config_data = json.load(json_data)

问题3: 如何获取命令行的输出结果

比如想获得 docker 容器的id, 可以用如下命令

docker ps -aqf name=jenkins

用 subprocess.check_output 方法就可以获取输出， 以下面这个函数为例

def get_container_id(container_name):
    str_filter = "-aqf name=%s" % container_name;
    arr_cmd = ["docker", "ps", str_filter]
    container_id = subprocess.check_output(arr_cmd).strip()
    return container_id

问题4: fab put error: paramiko.ssh_exception.SSHException: Channel closed

解决方法:

• 编辑 /etc/ssh/sshd_config:
  

  
   vi /etc/ssh/sshd_config
   

• 加上一行 Subsystem sftp internal-sftp
  

  
   Port 22
   Protocol 2
   LogLevel INFO
   X11Forwarding no
   MaxAuthTries 4
   IgnoreRhosts yes
   HostbasedAuthentication no
   PermitRootLogin yes
   PermitEmptyPasswords no
   PermitUserEnvironment no
   Ciphers aes128-ctr,aes192-ctr,aes256-ctr
   ClientAliveInterval 600
   Banner /etc/issue
   Subsystem sftp internal-sftp
   

• 保存并重启 SSH server:

    service sshd restart 

参考链接

• Fabric site
• Fabric Tutorial
• Fabric options