威胁情报数据采集

将公开的威胁情报数据采集做数据分析和内网安全问题 排查。情报获取只要可以提供全量数据的接口
情报一：
http://www.freebuf.com/sectool/159648.html
踩坑：

critical-stack-intel list
critical-stack 14:19:19 [INFO] Pulling feed list from the Intel Marketplace.

   ID  |                                    NAME                                     | LAST UPDATED | INDICATOR COUNT  
+------+-----------------------------------------------------------------------------+--------------+-----------------+
  217  | volexity.com-Wekby-Adobe-Flash-Exploit-IOCs                                 | -            | 0                
  216  | volexity.com-Virtual-Private-Keylogging-report-IOCs                         | -            | 0                
  215  | team-cymru.org-Poseidon-IOCs                                                | -            | 0                
  214  | talos-Angler-Exposed-2015-10-06-IOCs                                        | -            | 0                
  213  | symantec.com-Black-Vine-IOCs                                                | -            | 0                
  212  | symantec.com-Cadelspy-and-Remexi-Backdoor-IOCs                              | -            | 0                
  211  | shadowserver.org-The-Italian-Connection-IOCs                                | -            | 0                
  210  | secureworks.com-Point-of-Sale-Malware-Threats                               | -            | 0                
  207  | pan-unit42-Piano-Gh0st-IOCs                                                 | -            | 0                
  206  | pan-unit42-Lotus-Blossom-IOCs                                               | -            | 0                
  205  | pan-unit42-Dridex-2015-10-01-IOCs                                           | -            | 0                
  204  | morphick.com-BernhardPOS-IOCs                                               | -            | 0                
  203  | checkpoint.com-Rocket-Kitten:-A-Campaign-with-9-Lives-(2015-11-09)          | -            | 0                
  202  | anubisnetworks.com-Dridex-Botnet-Report-(2015-10-15)                        | -            | 0                
  201  | volexity.com-Wekby-Adobe-Flash-Exploit-IOCs                                 | -            | 0                
  200  | volexity.com-Virtual-Private-Keylogging-report-IOCs                         | -            | 0                
  199  | team-cymru.org-Poseidon-IOCs                                                | -            | 0                
  198  | talos-Angler-Exposed-2015-10-06-IOCs                                        | -            | 0                
  197  | symantec.com-Black-Vine-IOCs                                                | -            | 0                
  196  | symantec.com-Cadelspy-and-Remexi-Backdoor-IOCs                              | -            | 0                
  195  | shadowserver.org-The-Italian-Connection-IOCs                                | -            | 0                
  194  | secureworks.com-Point-of-Sale-Malware-Threats                               | -            | 0                
  192  | pan-unit42-Piano-Gh0st-IOCs                                                 | -            | 0                
  191  | pan-unit42-Lotus-Blossom-IOCs                                               | -            | 0                
  190  | pan-unit42-Dridex-2015-10-01-IOCs                                           | -            | 0                
  189  | morphick.com-BernhardPOS-IOCs                                               | -            | 0                
  188  | checkpoint.com-Rocket-Kitten:-A-Campaign-with-9-Lives-(2015-11-09)          | -            | 0                
  187  | anubisnetworks.com-Dridex-Botnet-Report-(2015-10-15)                        | -            | 0                
  186  | volexity.com-Wekby-Adobe-Flash-Exploit-IOCs                                 | -            | 0                
  185  | volexity.com-Virtual-Private-Keylogging-report-IOCs                         | -            | 0                
  184  | team-cymru.org-Poseidon-IOCs                                                | -            | 0                
  183  | talos-Angler-Exposed-2015-10-06-IOCs                                        | -            | 0                
  182  | symantec.com-Black-Vine-IOCs                                                | -            | 0            

critical-stack-intel pull
critical-stack 14:19:40 [ERROR] --- NOTICE ----------
critical-stack 14:19:40 [ERROR] Unable to locate bro or configure permissions properly.
critical-stack 14:19:40 [ERROR] Unable to add sudoers access for bro binary.
critical-stack 14:19:40 [INFO] If you have a custom setup you can add your paths manually.
critical-stack 14:19:40 [INFO] $ sudo critical-stack-intel config --set bro.path=/my/path/bro
critical-stack 14:19:40 [INFO] $ sudo critical-stack-intel config --set bro.include.path=/my/path/local.bro
critical-stack 14:19:40 [INFO] $ sudo critical-stack-intel config --set bro.broctl.path=/my/path/broctl
critical-stack 14:19:40 [ERROR] --- NOTICE ----------

很明显的数据并没有采集到，还报了一堆错，解决无果，后续厂商给答复了再更新。
情报二：
http://threatcrowd.blogspot.gr/2016/02/crowdsourced-feeds-from-threatcrowd.html 每小时 更新一次
获取方式：
https://www.threatcrowd.org/feeds/domains.txt
https://www.threatcrowd.org/feeds/ips.txt
https://www.threatcrowd.org/feeds/hashes.txt
页面做了简单的防DDOS,直接爬虫没法跳转。
selenium简单处理了一下:拿到数据，入库

# -*- coding: utf-8 -*-
from selenium import webdriver
from time import sleep
import sys
reload(sys)
sys.setdefaultencoding('utf8')
options = webdriver.ChromeOptions()
browser = webdriver.Chrome(executable_path='C:\Python27\Scripts\chromedriver.exe',chrome_options= options)  #mac系统的话chromedriver()放到 usr/local/bin/ 下面就可以 不需要禁用sip
def read(iplist):
    xx=iplist.split("\n")
    for i in range(len(xx)):
        if i==0:
            xxxx=xx[i].split(">")[-1]
            print xxxx
        elif i ==len(xx)-1:
            xxx = xx[i].split("<")[0]
            print xxx
        else:
            print i
            print xx[i]
def main():
    browser.get('https://www.threatcrowd.org/feeds/ips.txt')
    sleep(10)
    htmlstart1 = browser.page_source
    read(htmlstart1)
if __name__ == '__main__':
  main()

最后入库搞定
数据源三：
来源：https://github.com/scu-igroup/Ti_Collector
直接使用吧。后续持续更新。