saltstack结合Elasticsearch来做salt执行结果展现
salt虽然好用但是机器管理的越来越多，通过cli的结果输出方式查看执行结果越来越多不能满足我的需求，而且作为一个推动运维自动化的攻城狮，使用这种人眼查看执行结果的方式简直土到掉渣！虽然别人看起来逼格很高，但谁累谁知道。。。由于以上原因，给各位推荐一种逼格更高的结果查看方式：
salt returners
先来看一下官方结构图：
Send data returned by Salt Minions to another system, such as a database. Returners can run on the Salt Minion or on the Salt Master.
这个图上画的是redis，其实官方给出的returners有很多很多，这次为了装逼！哦不对，为了更好地筛选和过滤salt的执行结果，我选用了elasticsearch+kibana这种展现方式，好处是es提供一个准实时的全文检索引擎，这样我们可以实时的想搜什么搜什么，多维度过滤，而kibana提供的是便捷的用户界面和帅的一逼的画图界面！
但是，这两者是怎么结合起来的呢，很简单，用一个es支持salt也支持的数据中转站就行了，我选择的是kafka，当然你也可以用rsyslog，redis，等等，任君选择
下面就以 salt-minion -> kafka -> logstash -> elasticseach <- kibana <- 你的浏览器 这条主线进行说明，有纰漏的地方欢迎在评论里指正，谢谢！
现在张成果图镇楼：
下面正时开始：
1. salt-minion端的配置：
1.1 修改minion配置文件，这对于会用salt的我们来说改1w台机器就是分分钟的事儿嘛，但注意，改完要重启minion进程。
/etc/salt/minion：
#kafka broker主机，可以配置多个
returner.kafka.hostnames:
- "10.64.0.1"
- "10.64.0.2"
- "10.64.0.3"
#kafka topic名称
returner.kafka.topic: 'saltstack-topic'1.2 修改官方returner文件源代码，此处高深莫测，先按下不表
2.kafka 几乎不用配置，正常能用即可，顺便推荐一下kafka manager真好用，从此再也不用记命令了。
https://github.com/yahoo/kafka-manager
3.logstash 作为刚才minion发送到kafka数据的消费者，拿到kafka的数据后按json格式存到es中
3.1 logstash.conf：
input {
kafka {
#zk 连接串
zk_connect => '10.64.0.1:2181,10.64.0.2:2181,10.64.0.3:2181,10.64.1.174:2181,10.64.1.175:2181'
#与刚才配置的topic一致
group_id => 'logstash-saltstack'
topic_id => 'saltstack-topic'
consumer_id => 'logstash-saltstack-consumer-{{ grains["id"] }}'
consumer_threads => 1
queue_size => 200
codec => plain
}
}
filter {
mutate {
add_field => { "fromAgent" => "logstash-saltstack-{{ grains["id"] }}" }
}
json {
source => "message"
}
mutate {
remove_field => [ "message" ]
convert => { "fun_args" => "string" }
rename => { "__run_num__" => "run_num" }
}
}
output {
elasticsearch {
#输出到es
hosts => ["10.64.0.1:9200","10.64.0.2:9200","10.64.0.3:9200"]
index => "saltstack-%{+YYYY.MM}"
flush_size => 3000
idle_flush_time => 2
workers => 2
}
#stdout { codec => rubydebug }
}
这里有个小坑，顺便解释一下刚才按下不表的为啥非要改源码：es收数据的时候会把json全都拆成嵌套的kv格式存储，啥意思呢，请看图：
这是两个命令执行的不同返回，问题在哪儿呢？问题就在10.64.0.1这个key的value，一会儿是boolen型一会儿是nested型，这样es就没法办了，一个字段的类型实在mapping或者第一次创建数据时确定的，不能随便更改，这样就造成了es不能接收数据的问题。
怎么解决？开始我尝试了修改logstash的配置，不过那个实在不灵活。而且你注意看第二个salt执行的结果，是在一个大json里面的，这样我搜索起来匹配上的话也是一整条数据，还是不方便我看结果。
针对以上两点我还是觉得小改一下salt的returners，修改的地方如下（每台minion都要改，不过这次不用重启了）：
/usr/lib/python2.6/site-packages/salt/returners/kafka_return.py：
def returner(ret):
'''
Return information to a Kafka server
'''
if __salt__['config.option']('returner.kafka.topic'):
topic = __salt__['config.option']('returner.kafka.topic')
conn = _get_conn(ret)
producer = SimpleProducer(conn)
if ret["return"] == True or ret["return"] == False:
producer.send_messages(topic, json.dumps(ret))
else:
for retKey in ret["return"]:
myRet = {}
myRet["id"] = ret["id"]
myRet["fun"] = ret["fun"]
myRet["fun_args"] = ret["fun_args"]
myRet["jid"] = ret["jid"]
myRet["retcode"] = ret["retcode"]
for key in ret["return"][retKey]:
myRet[key] = ret["return"][retKey][key]
retKeys = retKey.split("_|-")
for i in range(0,len(retKeys)):
myRet["key"+str(i)] = retKeys[i]
producer.send_messages(topic, json.dumps(myRet))
_close_conn(conn)
else:
log.error('Unable to find kafka returner config option: topic')至此我们就应该能在es里看到每个动作一条记录的salt结果了。
4. es 还是没啥可配置的，正常运转就可以
5. kibana
首先，我用的是kibana 3，这个需要es <2.0版本的配合，不要赶新潮装es2，这样就用不了我的模板啦
5.1 载入kibana模板
把下面的内容存成一个文件，然后在kibana中选择载入，然后赶紧跑个salt任务试试
神马？没有，是这样，我忘了跟你说以后跑任务的时候要加个参数才能在es里看到哦：
salt 你要执行的命令 --return kafka
这样你就得到了开始看到的那个激动人心的逼格界面。 我有啥没说清楚的欢迎在下面评论里写明，我会尽力解答，谢谢！
{
"title": "Saltstack returns",
"services": {
"query": {
"idQueue": [
1
],
"list": {
"0": {
"query": "result:\"true\"",
"alias": "成功",
"color": "#7EB26D",
"id": 0,
"pin": false,
"type": "lucene",
"enable": true
},
"4": {
"id": 4,
"color": "#E24D42",
"alias": "失败",
"pin": false,
"type": "lucene",
"enable": true,
"query": "result:\"false\""
}
},
"ids": [
0,
4
]
},
"filter": {
"idQueue": [
1
],
"list": {
"0": {
"type": "time",
"field": "@timestamp",
"from": "now-1h",
"to": "now",
"mandate": "must",
"active": true,
"alias": "",
"id": 0
}
},
"ids": [
0
]
}
},
"rows": [
{
"title": "分类统计",
"height": "367px",
"editable": true,
"collapse": true,
"collapsable": true,
"panels": [
{
"error": false,
"span": 4,
"editable": true,
"type": "terms",
"loadingEditor": false,
"field": "type",
"exclude": [],
"missing": false,
"other": false,
"size": 50,
"order": "count",
"style": {
"font-size": "10pt"
},
"donut": false,
"tilt": false,
"labels": true,
"arrangement": "horizontal",
"chart": "table",
"counter_pos": "below",
"spyable": true,
"queries": {
"mode": "all",
"ids": [
0,
1,
2
]
},
"tmode": "terms",
"tstat": "total",
"valuefield": "",
"title": "type"
},
{
"error": false,
"span": 4,
"editable": true,
"type": "terms",
"loadingEditor": false,
"field": "host.raw",
"exclude": [],
"missing": false,
"other": false,
"size": 11,
"order": "count",
"style": {
"font-size": "10pt"
},
"donut": false,
"tilt": false,
"labels": true,
"arrangement": "horizontal",
"chart": "table",
"counter_pos": "none",
"spyable": true,
"queries": {
"mode": "all",
"ids": [
0,
1,
2
]
},
"tmode": "terms",
"tstat": "total",
"valuefield": "",
"title": "host"
},
{
"error": false,
"span": 4,
"editable": true,
"type": "terms",
"loadingEditor": false,
"field": "tags.raw",
"exclude": [],
"missing": false,
"other": false,
"size": 50,
"order": "count",
"style": {
"font-size": "10pt"
},
"donut": false,
"tilt": false,
"labels": true,
"arrangement": "horizontal",
"chart": "pie",
"counter_pos": "none",
"spyable": true,
"queries": {
"mode": "all",
"ids": [
0,
1,
2
]
},
"tmode": "terms",
"tstat": "total",
"valuefield": "",
"title": "tags"
}
],
"notice": false
},
{
"title": "Graph",
"height": "150px",
"editable": true,
"collapse": false,
"collapsable": true,
"panels": [
{
"span": 12,
"editable": true,
"group": [
"default"
],
"type": "histogram",
"mode": "count",
"time_field": "@timestamp",
"value_field": null,
"auto_int": true,
"resolution": 200,
"interval": "30s",
"fill": 3,
"linewidth": 3,
"timezone": "browser",
"spyable": true,
"zoomlinks": true,
"bars": true,
"stack": true,
"points": false,
"lines": false,
"legend": true,
"x-axis": true,
"y-axis": true,
"percentage": false,
"interactive": true,
"queries": {
"mode": "all",
"ids": [
0,
4
]
},
"title": "Events over time",
"intervals": [
"auto",
"1s",
"1m",
"5m",
"10m",
"30m",
"1h",
"3h",
"12h",
"1d",
"1w",
"1M",
"1y"
],
"options": true,
"tooltip": {
"value_type": "cumulative",
"query_as_alias": true
},
"annotate": {
"enable": false,
"query": "*",
"size": 20,
"field": "_type",
"sort": [
"_score",
"desc"
]
},
"pointradius": 5,
"show_query": true,
"legend_counts": true,
"zerofill": true,
"derivative": false,
"scale": 1,
"grid": {
"max": null,
"min": 0
},
"y_format": "none"
}
],
"notice": false
},
{
"title": "Events",
"height": "350px",
"editable": true,
"collapse": false,
"collapsable": true,
"panels": [
{
"title": "All events",
"error": false,
"span": 12,
"editable": true,
"group": [
"default"
],
"type": "table",
"size": 100,
"pages": 5,
"offset": 0,
"sort": [
"@timestamp",
"desc"
],
"style": {
"font-size": "9pt"
},
"overflow": "min-height",
"fields": [
"@timestamp",
"id",
"fun",
"fun_args",
"key0",
"key1",
"key2",
"key3",
"result"
],
"highlight": [],
"sortable": true,
"header": true,
"paging": true,
"spyable": true,
"queries": {
"mode": "all",
"ids": [
0,
4
]
},
"field_list": true,
"status": "Stable",
"trimFactor": 300,
"normTimes": true,
"all_fields": false,
"localTime": true,
"timeField": "@timestamp"
}
],
"notice": false
}
],
"editable": true,
"failover": false,
"index": {
"interval": "month",
"pattern": "[saltstack-]YYYY.MM",
"default": "NO_TIME_FILTER_OR_INDEX_PATTERN_NOT_MATCHED",
"warm_fields": true
},
"style": "dark",
"panel_hints": true,
"pulldowns": [
{
"type": "query",
"collapse": true,
"notice": false,
"query": "*",
"pinned": true,
"remember": 10,
"enable": true
},
{
"type": "filtering",
"collapse": true,
"notice": false,
"enable": true
}
],
"nav": [
{
"type": "timepicker",
"collapse": false,
"notice": false,
"status": "Stable",
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
],
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"timefield": "@timestamp",
"now": true,
"filter_id": 0,
"enable": true
}
],
"loader": {
"save_gist": false,
"save_elasticsearch": true,
"save_local": true,
"save_default": true,
"save_temp": true,
"save_temp_ttl_enable": true,
"save_temp_ttl": "30d",
"load_gist": true,
"load_elasticsearch": true,
"load_elasticsearch_size": 20,
"load_local": true,
"hide": false
},
"refresh": false
}