二、爬取伯乐在线

1. 创建main函数，调试scrapy

-- coding: utf-8 --

author = 'wujiee'

from scrapy.cmdline import execute

import sys
import os

sys.path.append(os.path.dirname(os.path.abspath(file)))
execute(["scrapy","crawl","jobbole"])

cmd命令中，运行scrapy crawl jobbole
如果出错，则安装如下包pip install pypiwin32

2.xpath用法
articleTitle = response.xpath('//[@id="post-112760"]/div[1]/h1/text()') --text()可以提取标签里的内容
tagList = response.xpath('//[@id="post-112760"]/div[2]/p/a/text()').extract() --extract()提取两个a标签的内容，组成一个数据列表
tags = ",".join(tagList) --join()函数拼接字符串，参数为list列表类型