用Flask写个爬虫玩

昨天彭老师和我说，希望我能和大家分享一下Flask这一块。碰巧最近在做flask的开发工作，手头上的这个工程项目可以拿出来和大家分享一下。

首先看一下我这个项目的结构

微信截图_20170603120704.png

app文件夹里是放功能模块的，Login是我写的登录模块，crawler是爬取单个页面修改网页源码返回前端的功能模块。其他的那些模块，是其他同事写的。我也不造他们写的什么。templates按理来说应该放的是网页的模板，但由于我们的开发工作是前后端分离的。所以里面放的只是邮件内容的模板。可能部分童鞋不理解什么是前后端分离。

微信截图_20170603122048.png

微信截图_20170603122107.png

上面两张图网站挺漂亮吧？但是，跟我的项目一毛钱关系没有。我整个flask项目跑起来。。。。。。连一个网页都木有。。。。。。那我这个项目是干什么用的？

我的这个Flask项目是给前端提供接口连接，返回数据给前端使用。前端做的是展示工作，我这里是后端实现的是功能。
crawler和·sportSpider`这个项目的两个主要功能。crawler是来爬取预览页，修改样式后在前端那里可以选择要爬取的元素，然后保存到数据库。sportSpider则调用后台scrapy爬虫项目，正式把爬虫跑起来！

直接看我写的那个crawler模块

# -*- coding: utf-8 -*-
from flask import Flask
from flask_cors import *
from . import crawler
from flask import Flask, jsonify,request,make_response,flash,redirect,url_for
from flask_login import login_user, logout_user, login_required, \
    current_user
from lxml import etree
import json
from bs4 import BeautifulSoup
import requests
import re

@crawler.route('/', methods=['GET'])
@cross_origin()
#@login_required
def Crawler():
    #url = request.form['url']
    url = request.args['url']
    print url
    headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0',
               'Accept - Language': 'zh - CN, zh;q = 0.8, en - US;q = 0.5, en;q = 0.3',
               }
    response = requests.get(url).text
    soup= BeautifulSoup(response,'html.parser')

    new_tag = soup.new_tag("style")
    content = """           .io-cursor-not-allowed-CHFG {
              cursor: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAeCAYAAAA/xX6fAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAyhpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuNi1jMTExIDc5LjE1ODMyNSwgMjAxNS8wOS8xMC0wMToxMDoyMCAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RSZWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiIHhtcDpDcmVhdG9yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENDIDIwMTUgKE1hY2ludG9zaCkiIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6NzMxNTk5N0RFREJGMTFFNUI1RDNERkQxQUFDMDRERjMiIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5kaWQ6NzMxNTk5N0VFREJGMTFFNUI1RDNERkQxQUFDMDRERjMiPiA8eG1wTU06RGVyaXZlZEZyb20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDo3MzE1OTk3QkVEQkYxMUU1QjVEM0RGRDFBQUMwNERGMyIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDo3MzE1OTk3Q0VEQkYxMUU1QjVEM0RGRDFBQUMwNERGMyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/PqJDRkIAAAL2SURBVHjavJZPaNJhGMdf/8w8bEJTUEg0sEMTNrzoLHYYUQe3IHbYwDp0Egq6OmKVVods7qDt1CEYBF1slzrIDuLBwxoiO0iDajCGIjEZSCApoj49z4vvcPn7rXTb74Hvz9/v8X2fz+993+d5lAHAa6aQIYtfQCkoB1arVdja2lIEyoGVSgUMBoMi0CMg3SsBPQZUAtoDFNBcLkfQZ4oASSaTCfL5PEEfKwI8L+iJwPOAElDb7dDpdKzRaDCfz8c8Hg/3ZTIZNj4+HqHBKpXq9MkkVuhwOGB5eZnfj42NQbvd/oWQF6jnXbp0JltKsGKxCLVa7bfRaOTQdDpNW3nnzJOmXq9zGNpDVGxxcZED5+bmyLdxHs27hXrQcVzZ29tr41mBRqOhF2mjz3FCDMoB2oV3qDTqC+oj6hHKLAe895dzY3Z2lq8yGo3SKldkYLdQ3+UyHFVFPUVpjgEl3uJ2NpuFZDIJrVaric9vJWD3US0aPjExAaurq7C5uck7VCKRgIWFBVCr1QL8SUDlgGpUFvUKZZWAXUM1aNsjkQhlM0gZvYDFYhHQFVngP0yFytHccDjMAxNwbW0NZmZmYGpqCpaWluDw8JB/t729DXq9nihN1NVBgNdFzWKD4EH9fn/P+dlsNpH5EAwGhf/NIMCXNDkUCvFg6+vrsm1xfn6ej9nZ2RG+Hz2t7T/sMl0wUfiD1Wpl+/v70nuvUvFPp9MpWqZd1FE/phM9l2xkZITZ7XbZwZjlDOuZS2Rqv8ASXXZ3d/kDlg/DmpUc6Ha7GZYIK5VKDFsmuX4O0p0oOni9Xn4+GAhcLlfP+eEOiL8qEI/Hhf/9IElzAVWkAFTgZNT8A4EAjI6OwtDQEExPTx/BDg4OumvxxiBAsrsUYHh4GFKpFMhZuVyGyclJAfs8aOELi9N8rVbLV0ctrdlsclChUIBYLAZms1nAvqEunhZIOf8EVRPnRr2z01W6lUSZTmzefZoNFUV9RdU7DZ0y+QPqptTP0x8BBgCoKq8fjjt0ZAAAAABJRU5ErkJggg==) 0 0, auto !important;
            }
    .io-cursor-add-CHFG {
              cursor: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAeCAYAAAA/xX6fAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAyhpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuNi1jMTExIDc5LjE1ODMyNSwgMjAxNS8wOS8xMC0wMToxMDoyMCAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RSZWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiIHhtcDpDcmVhdG9yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENDIDIwMTUgKE1hY2ludG9zaCkiIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6NzMxNTk5NzlFREJGMTFFNUI1RDNERkQxQUFDMDRERjMiIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5kaWQ6NzMxNTk5N0FFREJGMTFFNUI1RDNERkQxQUFDMDRERjMiPiA8eG1wTU06RGVyaXZlZEZyb20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDo3MzE1OTk3N0VEQkYxMUU1QjVEM0RGRDFBQUMwNERGMyIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDo3MzE1OTk3OEVEQkYxMUU1QjVEM0RGRDFBQUMwNERGMyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/PlbAVHoAAALUSURBVHjavJVPaBNBFIffbtMUKwY0OVQMtjH1EDSnQrHoxYO0ibZS8eTBHnpp/Ytii0IVe1ELgrSCeBD1ojnoQdqDRUrUoBBIvAQEFWsCaa3G2KCmaUzZrL+ZJmWz3dYkTXfg25l5M7tf5u3shGRZvkE6Fbj4RdZLyoXJZFL2+/26SLkwkUjIJpNJF+mSkLX1kBYI9ZAuE+alwWCQSS/rImRYLBY5FAox6UVdhOslXVW4HlImNCgDRqORMpkMuVwuam5u5jGfz0dOp/M6mywIwto3U36FdrtdHhoa4m2HwyFns9lfkAyCqwq2VSSlTBaNRuX5+fmU2WzmUq/Xy1J5uOKbJp1OcxlKL7jV39/PhZ2dnSw2XmmhgIuE9km8n7toN4bD4U9YsSCKIkUiEdlqte7E2KTWA/aONLE9cBC0gx1gA5gCr8GTt2fefVcLRdTHmYwFUH+22Wwv3G43SZJEHo9HQLhnBdkBVO/BM9AN9oM94Ci4DSYxZwBUKe8TNJZ9KBAIjMXjcWptbZWw0nv4IT0qWReq+0AsIpOj4AhWK/GUagjZQ/xgAtyBbEola8mlrLqE13cTwj5N4WoFMjY/AJqWPbFjhBq22OjC6FmKzH5RD7N9svvN6eAHscSN1qIlY8Wy0UJ1m7ZSdZVBa5i9x14q8h0oS9savgoXuxhKvKlB2Rlsu0a76pxLK+SpbR+hhewCb5942k2xZCw/vb4coVHZMdcuprEglhPzPIoGdVpLFk4rO1fGL1GNoSa3aYapfrON+sbYpgnzWCxZ8N3PlCP0gvP5zmzq59LAgrSYxvjcD5r5/VXr3pflbJqJ3NFVTnmoedIU8S0eQ/VIHa811pIoVFEqM0dZOaseHsOH35E/S0squPExqmF1PJVJUfLvHy3ZR9CV74hlpuccGADp/8x7DvbhRyZWPLxLTO92VKeAGzTmztdv4BV4ANGE+u/pnwADAJY2rJMWYmDjAAAAAElFTkSuQmCC) 0 0, auto !important;

            }
    .io-cursor-delete-CHFG {
              cursor: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAeCAYAAAA/xX6fAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAyhpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuNi1jMTExIDc5LjE1ODMyNSwgMjAxNS8wOS8xMC0wMToxMDoyMCAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RSZWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiIHhtcDpDcmVhdG9yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENDIDIwMTUgKE1hY2ludG9zaCkiIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6ODg4NEM2NzNFREIzMTFFNTkwNjNEMDE4MEY4NDJENjUiIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5kaWQ6ODg4NEM2NzRFREIzMTFFNTkwNjNEMDE4MEY4NDJENjUiPiA8eG1wTU06RGVyaXZlZEZyb20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDo4ODg0QzY3MUVEQjMxMUU1OTA2M0QwMTgwRjg0MkQ2NSIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDo4ODg0QzY3MkVEQjMxMUU1OTA2M0QwMTgwRjg0MkQ2NSIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pp/6/tcAAAJvSURBVHjavJVBiBJRGMffjNtEF6F0T0qbuQVCHiKQFrp0CNOKsPbcHrrs0l42yDq4UKcSoogiOkR1ykMF0R6KkKk8CXoSgooWDTd2Y10skDRhfP2/QWV2Uts3686D37w3z/fmN+973zwZ5/wGs6nApV+4XVJdWKvVeDabtUWqC6vVKnc6nbZIu0Jq2yFdJ7RD+o+wI83n8ySdt0VIuN1uXigUSHrFFuFWSQcKt0JKwhFjh6IorNlsskgkwkKhkN6XyWRYMBi8ToMlSdp8MnVW6Pf7eTKZ1NuBQIC3Wq1fkFwDVw14hhJSkpXLZV6v13+7XC5dqqoqhfL00JOm0WjoMpQZcDsej+vCWCxGfW+GLZRw0dC+gP15gPZ4sVj8ghVLsiyzUqnEvV7vPvy22OsBS2NnKQdOgFNgL9hB3eADeOb99uKHWSijPkcy6kD91efzvY1Go0zTNJZKpSR0T/eRHUP1EbwE58FRcBhMgrtgEWMSwGGcJ/VY9slcLrdQqVRYOBzWsNKHeJFpk2wK1SMgbyCSr8AZrFbTQ9pDSA/JgjS4D9mSSTbRDtk2ge27CeGlnsJBBTIanwOHBPOF8uSAp/T8kyw4ccKCjArt4wzb4B4Yy/FNfBURuowITtpjvNl1b44pB/f3Hbw6mWDa8lrndsyKUDHeyKM7mcMzOiCQDnNYhYXf153Ds7cY2670z5SVNePtshWhCi52H7j6U2TuOytJk24fXVbKE2EhPt4/qC5bkC1grmplhSR9iuqOwJTPYKqbaBbDMwcSoPGfca/BEbxkte/hLXjU7UY1C6JgvH2+roD34DFEafPf018BBgCuzILSC+9sGAAAAABJRU5ErkJggg==) 0 0, auto !important;
            }
    .light-green{border: 3px solid rgba(60, 147, 51, 1) !important;box-sizing: border-box;}"""
    soup.head.append(new_tag)
    soup.head.style.append(content)
    scriptTag = soup.new_tag("script",src='http://crawww.oss-cn-shenzhen.aliyuncs.com/console/build/js/transformer.js')
    #scriptContent = 'src="http://crawww.oss-cn-shenzhen.aliyuncs.com/console/build/js/transformer.js"'
    soup.body.append(scriptTag)
    #soup.body.script.append(scriptContent)
    res = soup.prettify()
    clsname = re.findall(r'class=".*?"', res)
    newclsname = []
    for i in clsname:
        if i not in newclsname:
            newclsname.append(i)
    for j in newclsname:
        css = re.findall(r'"(.*?)"', j)[0]
        res = res.replace(j, ' class="%s io-cursor-add-CHFG  light"' % css)
    file = open('crawler.html', 'w')
    file.write(res.encode('utf-8'))
    file.close()
    return res

这个API接口是个爬虫，传入url参数，就把网页的源码爬回来进行样式的修改，留作后续使用。

微信截图_20170603141815.png

来看看返回的页面

微信截图_20170603141905.png

这是修改过网页源码的页面，绿色的框框是在前端那里选择元素使用的，然后保存到数据库。之后前端那里点个确认按钮，就调用sportSpider这个接口，把后台爬虫激活，让爬虫正式跑起来。
这是sportSpider的部分代码

@sportSpiders.route('/run', methods=['POST'])
@cross_origin()
def run():
    # 接收userId，_id(爬虫id)
    userId = request.form['userId'] 
    _id = request.form['_id'] 
    #===========================================================================
    # 运行爬虫，记得配置日志文件目录
    #===========================================================================

    # 爬虫文件放在哪里？用requests写，爬虫文件为sportSpider.py，保存为_<_id>.out日志文件
    # os.system('nohup python C:\Users\lenovo\crawler {} 2>&1 > ~/logPath/{}_{}.out &'.format(_id, userId, _id))
    os.chdir('C:\Users\lenovo\crawler')#必须切换目录，不然爬虫跑不起来
    os.system('python scripts.py {} &'.format(userId))#执行命令，让爬虫启动
    # time.sleep(2)
    # process = os.popen('cat ~/Desktop/crawservice/crawwwservice/{}_{}.out'.format(userId, _id))

    # res = {'message':process.read(),'code':200}
    res = {'message':'run success','code':200}
    return json.dumps(res)

if __name__ == '__main__':
    print 'heeheh'

看看后台爬虫项目

微信截图_20170603144038.png

这里我就不使用 scrapy crawl myspider这条命令来运行爬虫，而是通过 scripts.py这个脚本让爬虫跑起来

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
import sys

args = sys.argv
_id= args[1]
print _id
process = CrawlerProcess(get_project_settings())
print get_project_settings()
name = ['myspider']
#for i in name:
#    process.crawl(i,userid='leon')
#    process.start()
process.crawl('myspider',_id)
process.start()

需要爬取的数据已经在数据库了，通过调用这个接口传入参数，我们的后台爬虫就可以正式跑起来了！

微信截图_20170603142820.png

看看终端

微信截图_20170603143042.png

scrapy爬虫就这样通过flask跑起来了！
不过后面的爬虫数据爬取状态还没写完，所以无法展示效果。

暂时，就到此为止吧

这不是一个教程文章，这篇文章的目的主要是分享一下用flask来运行爬虫项目的思路，所以很多细节的东西，包括怎么使用flask的我也不写了，网上都有教程的。

用Flask写个爬虫玩

你可能感兴趣的:(用Flask写个爬虫玩)