爬取10万商品数据

出现的问题：

• 当爬取交易地点，使用代码

时，爬取的结果是 ['交易地点：']，而不是我想要的‘地点－地点’形式

解决方法

将selector改为

soup.select('#wrapper > div.content.clearfix > div.leftBox > div > div > ul > li:nth-of-type(3) > a)

如果仅仅是用

soup.select('#wrapper > div.content.clearfix > div.leftBox > div > div > ul > li:nth-of-type(3) > a)[0].stripped_strings

那么结果也只是网站中出现的第一个地点，而不是全部
所以，采用map()函数遍历

• 在爬取过程中出现了

requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(54, 'Connection reset by peer')", ConnectionResetError(54, 'Connection reset by peer'))
```这种错误
在网上查了一下，是属于[python requests接收chunked编码问题](http://blog.csdn.net/wangzuxi/article/details/40377467)
但是网上给出的解决方法太专业，我一入门汉瞬间懵逼了
#####解决方法
后来我注意到在网页的Request Headers中有一个Accept-Encoding,应该是关于编码问题的，所以就在在requests.get中添加了
```headers = { 'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4)AppleWebKit/537.36(KHTML,likeGecko)Chrome/44.0.2403.157 Safari/537.36',    'Connection':'keep-alive','Accept-Encoding':'gzip, deflate'}```然后就顺利爬取了
- 后来爬取到5万多条时出现了```requests.exceptions.ConnectionError: None: Max retries exceeded with url: /qitawupin/o111/ (Caused by None)```错误提示
######解决方法
使用代理ip


#下面是代码部分
- 爬取分类链接

import requests
from bs4 import BeautifulSoup
first_url = 'http://bj.ganji.com/wu/'
base_url = 'http://bj.ganji.com'

http://bj.ganji.com/jiaju/

def get_second_url(url):
web_data = requests.get(url)
soup = BeautifulSoup(web_data.text, 'lxml')
second_urls = soup.select('dl.fenlei dt a')
for second_url in second_urls:
whole_second_url = base_url + second_url.get('href')
print(whole_second_url)

将得到的结果赋值给whole_second_url 

- 爬取列表页

mport requests,time,pymongo,random
from bs4 import BeautifulSoup
client = pymongo.MongoClient('localhost',27017)
ganji = client['ganji']
whole_third_url = ganji['whole_third_url']
item_info = ganji['item_info']
headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36', 'Connection':'keep-alive','Accept-Encoding':'gzip, deflate'}
proxy_list = ['http://117.177.250.151:8081', 'http://111.85.219.250:3129', 'http://122.70.183.138:8118',]
proxy_ip = random.choice(proxy_list)
proxies = {'http':proxy_ip}
def get_third_url(whole_second_url,pages):
whole_url = '{}o{}/'.format(whole_second_url,str(pages))
web_data = requests.get(whole_url,headers = headers,proxies = proxies)
# time.sleep(5)
soup = BeautifulSoup(web_data.text, 'lxml')
if soup.find_all('a',{'class':'next'}):
for link in soup.select('li.js-item a.ft-tit'):
third_url = link.get('href')
whole_third_url.insert_one({'url':third_url})
#print(third_url)
else:
pass

- 爬取详情页具体信息

def get_item_info(url):
web_data = requests.get(url)
soup = BeautifulSoup(web_data.text, 'lxml')
title = soup.select('h1.title-name')[0].text if soup.find_all('h1',{'class':'title-name'}) else None
#这里考虑到有删除页和转转的商品页（和一般的商品页不一样），根据观察，删除页和转转页的商品标题的html不一样，所以以标题作为判断标准
if title == None:
pass
else:
time = list(soup.select('i.pr-5')[0].stripped_strings) if soup.find('i',{'class':'pr-5'}) else None
type = soup.select('#wrapper > div.content.clearfix > div.leftBox > div:nth-of-type(3) > div > ul > li:nth-of-type(1) > span > a')[0].text if soup.find_all('ul',{'class':'det-infor'}) else None
price = soup.select('i.f22.fc-orange.f-type')[0].text if soup.find_all('i',{'class':'f22 fc-orange f-type'}) else None
address = list(map(lambda x:x.text,soup.select('#wrapper > div.content.clearfix > div.leftBox > div > div > ul > li:nth-of-type(3) > a'))) if soup.find_all('li') else None
old_new = soup.select('ul.second-det-infor.clearfix > li:nth-of-type(2) > label')[0].text if soup.select('ul.second-det-infor.clearfix > li:nth-of-type(2) > label') else None
item_info.insert_one({'title':title, 'time':time, 'type':type, 'price':price, 'address':address, 'old_new':old_new})

    print(title,time,type,price,address,old_new)

- 开始爬取 

from multiprocessing import Pool
from get_second_url import whole_second_url
from get_third_url import get_third_url
from get_third_url import get_item_info
def get_all_links_from(whole_second_url):
for i in range(1,121):
get_third_url(whole_second_url,i)

if name == 'main':
pool = Pool()
pool.map(get_all_links_from,whole_second_url.split())

- 计数
使用以下程序对存储到数据库的数据进行计数

import time
from get_third_url import whole_third_url
while True:
print(whole_third_url.find().count())
time.sleep(5)