python爬虫学习--pixiv爬虫(2)--国际排行榜的图片爬取

之前用面向过程的形式写了一下pixiv爬虫的登录...

觉得还是面向对象好一些...

那就先把登录过程重写一下...

class Pixiv_Spider:

    def __init__(self):
        self.p_id = ''
        self.p_pw = ''

    def Login(self):                        #处理登录所需要的请求信息

        p_login_url = 'https://www.pixiv.net/login.php'
        
        data = {                                    #登录所要post的信息
                'mode':'login',
                'skip':1
                }

        data['pixiv_id'] = self.p_id                #传入登录id以及password
        data['pass'] = self.p_pw

        p_login_data = urllib.urlencode(data)

        p_login_header = {                          #头信息
                'accept-language':'zh-cn,zh;q=0.8',
                'referer':'https://www.pixiv.net/login.php?return_to=0',
                'user-agent':'mozilla/5.0 (windows nt 10.0; win64; x64; rv:45.0) gecko/20100101 firefox/45.0'
                }

        request = urllib2.Request(
                url = p_login_url,
                data = p_login_data,
                headers = p_login_header
                )
        try:
            cookie_file = 'cookie.txt'                  #生成cookie
            cookie = cookielib.MozillaCookieJar(cookie_file)
            opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie))         
            response = opener.open(request)             #登录
            cookie.save(ignore_discard = True,ignore_expires = True)
        except urllib2.URLError,e:
            if hasattr(e,"reason"):
                print "登录失败？？？",e.reason

ps = Pixiv_Spider()
ps.p_id = raw_input('请输入你的pixiv id:')
ps.p_pw = raw_input('请输入你的pixiv密码:')
ps.Login()

登录完成后就可以进行我们想要图片批量爬取了...

首先需要写一个方法利用一下前面登录的cookie...

def Cookie_Login(self):                         #读取之前登陆生成的cookie
    cookie_login = cookielib.MozillaCookieJar()
    cookie_login.load('cookie.txt',ignore_discard = True,ignore_expires = True)
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_login))
    return opener

为了以后的需要...

我们在这里写一个选项菜单...

方便以后添加更多的功能...

def Choice_Pixiv(self,opener):     #选择要跳转到的页面
    if (self.p_choice == '1'):
        try:
            p_page = opener.open(self.p_international_url)   #p_international_url = 'http://www.pixiv.net/ranking_area.php?type=detail&no=6'
            p_international = p_page.read().decode('utf-8')  #利用cookie登录后的页面
        except urllib2.URLError,e:
            if hasattr(e,"reason"):
                print "连接错误:",e.reason

这个时候就可以写今天的主体了国际榜的方法了...

这次需要一个非常厉害的东西 BeautifulSoup...它可以帮助我们在这里进一步提取html中的关键细节

def Pixiv_International(self,opener,p_international,dl_dir):
    soup = BeautifulSoup(p_international)
    for i in range(1,101):                    #已知pixiv国际榜的排名为100名，用for循环来完成
     get_information = str(soup.find(id=i))          #通过bs处理html将我们所需要的信息大体提取出来

在pixiv的国际榜中有单张上传的图片，多张上传的图片，还有一中漫画格式和动图

先来看一下他们在页面代码中是什么样子

单张图片：

<a class="work _work " href="member_illust.php?mode=medium&illust_id=56037267"> 

href中的就是我们正常浏览作品时的url的一部分

so...将它提取出来...

 result_url = re.search(re.compile('<.*?work\s_work\s".*?href="(.*?)">',re.S),get_information)

多张图片：

动图：

漫画多图：

同理还是提取出中间的href...

result_multiple = re.search(re.compile('<a.*?work\s_work\smultiple\s.*?href="(.*?)">',re.S),get_information)     #多图
result_video = re.search(re.compile('<a.*?work\s_work\sugoku-illust\s.*?href="(.*?)">',re.S),get_information)    #动图
result_manga_multiple = re.search(re.compile('<a.*?work\s_work\smanga\smultiple\s.*?href="(.*?)">',re.S),get_information)   #漫画多图

在当我们用re.search搜索不到的时候，他就会返回一个None...利用这一点我们就可以判断图片模式了

由于功力不足...我无法抓取动图...所以放弃动图...

其他的单图，多图和漫画多图在后面处理的时候不一样...所以这样写

if result_video == None:                                #判断是否是动图
    if result_manga_multiple == None:                   #判断是否为manga
        if result_multiple == None:                     #判断是否为多图
            p_url = 'http://www.pixiv.net/' + result_url.group(1)
        else:
            p_url = 'http://www.pixiv.net/' + result_multiple.group(1)
    else:
        p_url = 'http://www.pixiv.net/' + result_manga_multiple.group(1)
else:
    print "诶呀！这是张动图...无能为力啊...╮(╯▽╰)╭"

这下子我们就能拥有浏览这些图片的url了...

但是这个时候我们输出一下会发现有些和我们想象中的不一样啊（╯‵□′）╯︵┴─┴

http://www.pixiv.net/member_illust.php?mode=medium&illust_id=56039502

http://www.pixiv.net/member_illust.php?mode=medium&illust_id=56039502

经过仔细对比发现是url中间的&在前面的处理中被转义成&了...

我们再写一个工具类来帮助我们将他转换回来...

class Tools:

    remove = re.compile('amp;')

    def removesomething(self,x):
        x = re.sub(self.remove,"",x)
        return x.strip()

通过re.compile找出多出来的amp;，再用re.sub赋空字符就可以了...

想要调用这个类，在爬虫类Pixiv_Spider的__init__中增加这条代码就可以了

def __init__(self):
    self.tool = Tools()

经过一番折腾现在的代码变成这样了

if result_video == None:
    if result_manga_multiple == None:                   #判断是否为manga
        if result_multiple == None:                     #判断是否为多图
             print "报告！前方发现单张张图片..."
             p_url = self.tool.removesomething('http://www.pixiv.net/' + result_url.group(1))
        else:
            print "报告！前方发现多张图片..."
           p_url = self.tool.removesomething('http://www.pixiv.net/' + result_multiple.group(1))
    else:
        print "报告！前方发现多张图片..."
        p_url = self.tool.removesomething('http://www.pixiv.net/' + result_manga_multiple.group(1))
else:
     print "诶呀！前方这是张动图...无能为力啊...╮(╯▽╰)╭"

光url提取出来还不够，我还想保存这些图片的信息：标题，p站id,作者等等...

再来看一下刚刚的html...

标题：

p站id：

作者：

再写一个方法将这些信息打印到屏幕上并将它们以文件形式保存...

def Download_Data(self,i,get_information,p_url):
    #通过使用正则表达式再处理一遍经过bs处理的html代码，找到需要的信息(url,title,user)
    result_title = re.search(re.compile('<a href=".*?>(.*?)</a>',re.S),get_information)
    result_id = re.search(re.compile('<a class.*?illust_id=(.*?)">',re.S),get_information) 
    result_user = re.search(re.compile('<span class.*?>(.*?)</span>',re.S),get_information)    
    p_rank = str(i)                   #在屏幕上输出信息
    print "RANK #" + p_rank
    p_id = result_id.group(1)
    print "Pixiv ID:" + p_id
    p_title = result_title.group(1)
    print "Title:" + p_title
    p_user = result_user.group(1)
    print "User:" + p_user
    file_data = open('E:/pixivdata/' + dl_dir + '/pixiv_' + p_id + '.txt','w')     #创建信息文件
    massage = [                         #保存信息
           'rank:' + p_rank +'\n',
           'id:' + p_id + '\n',
           'title:' + p_title + '\n',
           'user:' + p_user + '\n',
           'url:' + p_url
    ]
    file_data.writelines(massage)
    file_data.close()
    print "pixiv信息保存成功"           #将信息以txt格式保存下来
   return p_id

墨迹了这么长时间，收集了足够的信息...现在可以写下载的部分了...

先看下网页...

单张图片：

多张图片(漫画和普通多图)：

经过观察我们发现多图的界面可以看到一共有多少张图片...这样子我们就可以先将多图的图片数提取出来...

soup = BeautifulSoup(opener.open(p_url))
result_pic_more = re.search(re.compile('</li><li>.*?\s(.*?)P</li>',re.S),str(soup.find_all("ul",class_="meta")))
print "报告！发现图片" + result_pic_more.group(1) + "张..."

点开多图...跳转到另一个页面...

通过查看网页代码发现他指向了这样一个url...

点进去发现这里就是我们想要的原图，后面的page控制了图片在多图中的顺序

http://www.pixiv.net/member_illust.php?mode=manga_big&illust_id=56039502&page=0

除了page以外...这个url和我们刚刚爬到的url还是有点区别...那就在Tools()里给它构造出一个一样的

http://www.pixiv.net/member_illust.php?mode=medium&illust_id=56039502

make_m = re.compile('mode=medium')

def make_big_url(self,x):
    x = re.sub(self.make_m,"mode=manga_big",x)
    return x.strip()

于是通过现有信息来进行爬取

for j in range(0,int(result_pic_more.group(1))):            
    make_url = self.tool.make_big_url(p_url)+'&page='+str(j)      #构造url
    m_soup = BeautifulSoup(opener.open(make_url))
    real_url = re.search(re.compile('<img.*?src="(.*?)"/>',re.S),str(m_soup.find_all("img")
    print '成功找到大图链接（ˉ﹃ˉ）...\n' + real_url.group(1)   #下载图片并保存
    d_url = opener.open(make-url)
   file_pic = open('E:/pixivdata/pixiv_' + p_id + '_' + str(j) + '.jpg','wb')
    file_pic.write(d_url.read)
    file_pic.close()

但是到程序到这里就会报错...

找来找去发现在当访问原图的url的时候浏览器会向服务器发送一个请求头...这个请求头与之前的请求头有区别的是它多了一个参数Referer...

如果在只访问原图url的时候...浏览器（我们的程序）并不会发送带Referer的请求头...服务器收到这个请求头但是不会回应...于是就悲催了...403

那么我们就给他一个Referer...

def Download_Request(self,opener,make_url,real_url):
    p_download_header = {                          #头信息
        'Accept-Language':'zh-CN,zh;q=0.8',
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0'
    }
    p_download_header['Referer'] = make_url        #将referer加入header，没有referer会显示403                
    download_request = urllib2.Request(
        url = real_url.group(1),
        headers = p_download_header
        )
    decode_url = opener.open(download_request)
    return decode_url.read()

不光是这里...细心的人可以发现我们提交的Referer和浏览器上看到的有点区别

我们还是在刚刚的Tools()里再写一个方法处理它

rmbig = re.compile('_big')

def removebig(self,x):
    x = re.sub(self.rmbig,"",x)
    return x.strip()

p_download_header['Referer'] = self.tool.removebig(make_url)

这下可以成功抓取多张图片了...

不过我在这里再给程序添加个小功能...就是将图片按照服务器上的文件格式进行保存...在Tool()中添加下面的方法...

def Pic_Type(self,real_url):                    #区分图片分辨率
    p_type = re.search(re.compile('png',re.S),real_url)
    if p_type == None:
        self.pic_type = 'jpg'
        return self.pic_type
    else:
        self.pic_type = 'png'
        return self.pic_type

这样我们多图下载的功能就完成了

for j in range(0,int(result_pic_more.group(1))):            
    make_url = p_url+'&page='+str(j)       #生成多张的url
    m_soup = BeautifulSoup(opener.open(make_url))
    real_url = re.search(re.compile('<img.*?src="(.*?)"/>',re.S),str(m_soup.find_all("img")))
    p_type = self.tool.Pic_Type(real_url.group(1))
    print '成功找到大图链接（ˉ﹃ˉ）...\n' + real_url.group(1)#下载图片并保存
    file_pic = open('E:/pixivdata/pixiv_' + p_id + '_' + str(j) + '.' + p_type,'wb')  
    file_pic.write(self.Download_Request(opener,make_url,real_url))
    file_pic.close()
    print '成功下载到本地(/≧▽≦)/...'

点开单图...使其最大化...用查看网页代码...

找到了单张图片大图的url...

http://i4.pixiv.net/img-original/img/2016/03/27/16/00/01/56037267_p0.png

和多图的差不多

soup = BeautifulSoup(opener.open(p_url))
real_url = re.search(re.compile('.*?data-src="(.*?)"',re.S),str(soup.find_all("img",class_="original-image")))
print '成功找到大图链接(ˉ﹃ˉ)...\n' + real_url.group(1)
p_type = self.tool.Pic_Type(real_url.group(1))
file_pic = open('E:/pixivdata/' + dl_dir + '/pixiv_' + p_id + '.' + p_type,'wb')                
file_pic.write(self.Download_Request(opener,make_url,real_url))
file_pic.close()
print '成功下载到本地(/≧▽≦)/...'

整理一下...下载的方法就写好了...

def Download_Pic(self,p_num,i,opener,p_url,p_id,dl_dir):
    if p_num == '1':
    soup = BeautifulSoup(opener.open(p_url))
    real_url = re.search(re.compile('.*?data-src="(.*?)"',re.S),str(soup.find_all("img",class_="original-image")))
    print '成功找到大图链接(ˉ﹃ˉ)...\n' + real_url.group(1)
    p_type = self.tool.Pic_Type(real_url.group(1))
    file_pic = open('E:/pixivdata/' + dl_dir + '/pixiv_' + p_id + '.' + p_type,'wb')
       file_pic.write(self.Download_Request(opener,p_url,real_url))
    file_pic.close()
    print '成功下载到本地(/≧▽≦)/...'
            
   if p_num == 'more':
       soup = BeautifulSoup(opener.open(p_url))
    result_pic_more = re.search(re.compile('</li><li>.*?\s(.*?)P</li>',re.S),str(soup.find_all("ul",class_="meta")))
    print "发现图片" + result_pic_more.group(1) + "张...⊙▽⊙"
    for j in range(0,int(result_pic_more.group(1))):
           make_url = self.tool.make_big_url(p_url)+'&page='+str(j)      #生成多张的url
           m_soup = BeautifulSoup(opener.open(make_url))
           real_url = re.search(re.compile('<img.*?src="(.*?)"/>',re.S),str(m_soup.find_all("img")))
           p_type = self.tool.Pic_Type(real_url.group(1))
           print '成功找到大图链接（ˉ﹃ˉ）...\n' + real_url.group(1)     #下载图片并保存
           file_pic = open('E:/pixivdata/' + dl_dir + '/pixiv_' + p_id + '_' + str(j) + '.' + p_type,'wb')  
           file_pic.write(self.Download_Request(opener,make_url,real_url))
           file_pic.close()
      print '成功下载到本地(/≧▽≦)/...'

写了这么多我们整理下就是我们想要的小程序了

#coding:UTF-8

__author__ = 'monburan'
__version__ = '0.10 only_international'

import os
import re
import urllib
import urllib2
import cookielib
from urllib2 import urlopen
from bs4 import BeautifulSoup

class Tools:

    remove = re.compile('amp;')
    rmbig = re.compile('_big')
    make_m = re.compile('mode=medium')
    
    def removebig(self,x):
        x = re.sub(self.rmbig,"",x)
        return x.strip()

    def removesomething(self,x):
        x = re.sub(self.remove,"",x)
        return x.strip()

    def make_big_url(self,x):
        x = re.sub(self.make_m,"mode=manga_big",x)
        return x.strip()

    def Pic_Type(self,real_url):                    #区分图片分辨率
        p_type = re.search(re.compile('png',re.S),real_url)
        if p_type == None:
            self.pic_type = 'jpg'
            return self.pic_type
        else:
            self.pic_type = 'png'
            return self.pic_type

class Pixiv_Spider:

    def __init__(self):
        self.tool = Tools()
        self.p_id = ''
        self.p_pw = ''
        self.p_choice = ''
        self.dl_dir = ''
        self.pic_type = ''
        self.p_international_url = 'http://www.pixiv.net/ranking_area.php?type=detail&no=6'     #国际排行榜url

    def Login(self):                        #处理登录所需要的请求信息
        p_login_url = 'https://www.pixiv.net/login.php'        
        data = {                                    #登录所要post的信息
                'mode':'login',
                'skip':1
                }
        data['pixiv_id'] = self.p_id                #传入登录id以及password
        data['pass'] = self.p_pw
        p_login_data = urllib.urlencode(data)
        p_login_header = {                          #头信息
                'accept-language':'zh-cn,zh;q=0.8',
                'referer':'https://www.pixiv.net/login.php?return_to=0',
                'user-agent':'mozilla/5.0 (windows nt 10.0; win64; x64; rv:45.0) gecko/20100101 firefox/45.0'
                }
        request = urllib2.Request(
                url = p_login_url,
                data = p_login_data,
                headers = p_login_header
                )
        try:
            cookie_file = 'cookie.txt'                  #生成cookie
            cookie = cookielib.MozillaCookieJar(cookie_file)
            opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie))         
            response = opener.open(request)             #登录
            cookie.save(ignore_discard = True,ignore_expires = True)
        except urllib2.URLError,e:
            if hasattr(e,"reason"):
                print "登录失败？？？",e.reason
    
    def Download_Request(self,opener,make_url,real_url):
            p_download_header = {                          #头信息
                'Accept-Language':'zh-CN,zh;q=0.8',
                'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0'
            }

            p_download_header['Referer'] = self.tool.removebig(make_url)        #将处理过的referer加入header，没有referer会显示403
                
            download_request = urllib2.Request(
                url = real_url.group(1),
                headers = p_download_header
                ) 
            decode_url = opener.open(download_request)
            return decode_url.read()

    def Cookie_Login(self):                         #读取之前登陆生成的cookie
            cookie_login = cookielib.MozillaCookieJar()
            cookie_login.load('cookie.txt',ignore_discard = True,ignore_expires = True)
            opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_login))
            return opener

    def Choice_Pixiv(self,opener):     #选择要跳转到的页面
        if (self.p_choice == '1'):
            try:
                p_page = opener.open(self.p_international_url)
                p_international = p_page.read().decode('utf-8')
                dl_dir = 'international'
                self.Pixiv_International(opener,p_international,dl_dir)
            except urllib2.URLError,e:
                if hasattr(e,"reason"):
                    print "连接错误:",e.reason

    def Pixiv_International(self,opener,p_international,dl_dir):
        soup = BeautifulSoup(p_international)
        os.mkdir(r'E:/pixivdata/' + dl_dir + '/')          #生成文件夹
        print "生成"+dl_dir+"目录成功！"
        for i in range(1,101):                              #已知pixiv国际榜的排名为100名，用for循环来完成
            get_information = str(soup.find(id=i))          #通过bs处理html将我们所需要的信息大体提取出来
            result_url = re.search(re.compile('<.*?work\s_work\s".*?href="(.*?)">',re.S),get_information)
            result_multiple = re.search(re.compile('<a.*?work\s_work\smultiple\s.*?href="(.*?)">',re.S),get_information)
            result_video = re.search(re.compile('<a.*?work\s_work\sugoku-illust\s.*?href="(.*?)">',re.S),get_information)
            result_manga_multiple = re.search(re.compile('<a.*?work\s_work\smanga\smultiple\s.*?href="(.*?)">',re.S),get_information)            
            if result_video == None:
                if result_manga_multiple == None:                   #判断是否为manga
                    if result_multiple == None:                     #判断是否为多图
                        p_num = '1'
                        p_url = self.tool.removesomething('http://www.pixiv.net/' + result_url.group(1))
                        print "报告！前方发现单张图片..."
                        p_id = self.Download_Data(i,get_information,p_url,opener,dl_dir)
                        self.Download_Pic(p_num,i,opener,p_url,p_id,dl_dir)
                    else:
                        p_num = 'more'
                        p_url = self.tool.removesomething('http://www.pixiv.net/' + result_multiple.group(1))
                        print "报告！前方发现多张图片..."
                        p_id = self.Download_Data(i,get_information,p_url,opener,dl_dir)
                        self.Download_Pic(p_num,i,opener,p_url,p_id,dl_dir)
                else:
                    p_num = 'more'
                    p_url = self.tool.removesomething('http://www.pixiv.net/' + result_manga_multiple.group(1))
                    print "报告！前方发现多张漫画..."
                    p_id = self.Download_Data(i,get_information,p_url,opener,dl_dir)
                    self.Download_Pic(p_num,i,opener,p_url,p_id,dl_dir)
            else:
                print "报告！前方这是张动图...无能为力啊...╮(╯▽╰)╭"

    def Download_Data(self,i,get_information,p_url,opener,dl_dir):
        #通过使用正则表达式再处理一遍经过bs处理的html代码，找到需要的信息(url,title,user)
        result_title = re.search(re.compile('<a href=".*?>(.*?)</a>',re.S),get_information)
        result_id = re.search(re.compile('<a class.*?illust_id=(.*?)">',re.S),get_information) 
        result_user = re.search(re.compile('<span class.*?>(.*?)</span>',re.S),get_information)        
        p_rank = str(i)
        p_id = result_id.group(1)
        p_title = result_title.group(1)
        p_user = result_user.group(1)
        print "RANK #" + p_rank + "\nPixiv ID:" + p_id + "\nTitle:" + p_title +"\nUser:" + p_user
        file_data = open('E:/pixivdata/' + dl_dir + '/pixiv_' + p_id + '.txt','w')     #创建信息文件
        massage = [                         #保存信息
            'rank:' + p_rank +'\n',
            'id:' + p_id + '\n',
            'title:' + p_title + '\n',
            'user:' + p_user + '\n',
            'url:' + p_url
            ]
        file_data.writelines(massage)
        file_data.close()
        print "报告！pixiv信息保存成功..."           #将信息以txt格式保存下来
        return p_id

    def Download_Pic(self,p_num,i,opener,p_url,p_id,dl_dir):
        if p_num == '1':
            soup = BeautifulSoup(opener.open(p_url))
            real_url = re.search(re.compile('.*?data-src="(.*?)"',re.S),str(soup.find_all("img",class_="original-image")))
            print '成功找到大图链接(ˉ﹃ˉ)...\n' + real_url.group(1)
            p_type = self.tool.Pic_Type(real_url.group(1))
            file_pic = open('E:/pixivdata/' + dl_dir + '/pixiv_' + p_id + '.' + p_type,'wb')                
            file_pic.write(self.Download_Request(opener,p_url,real_url))
            file_pic.close()
            print '成功下载到本地(/≧▽≦)/...'            
        if p_num == 'more':
            soup = BeautifulSoup(opener.open(p_url))
            result_pic_more = re.search(re.compile('</li><li>.*?\s(.*?)P</li>',re.S),str(soup.find_all("ul",class_="meta")))
            print "发现图片" + result_pic_more.group(1) + "张...⊙▽⊙"
            for j in range(0,int(result_pic_more.group(1))):
                make_url = self.tool.make_big_url(p_url)+'&page='+str(j)        #生成多张的url
                m_soup = BeautifulSoup(opener.open(make_url))
                real_url = re.search(re.compile('<img.*?src="(.*?)"/>',re.S),str(m_soup.find_all("img")))
                p_type = self.tool.Pic_Type(real_url.group(1))
                print '成功找到大图链接（ˉ﹃ˉ）...\n' + real_url.group(1)     #下载图片并保存
                file_pic = open('E:/pixivdata/' + dl_dir + '/pixiv_' + p_id + '_' + str(j) + '.' + p_type,'wb')  
                file_pic.write(self.Download_Request(opener,make_url,real_url))
                file_pic.close()
                print '成功下载到本地(/≧▽≦)/...'        

    def Program_Start(self):
        self.Login()
        opener = self.Cookie_Login()
        self.Choice_Pixiv(opener)

ps = Pixiv_Spider()
ps.p_id = raw_input('请输入你的pixiv id:')
ps.p_pw = raw_input('请输入你的pixiv密码:')
print ('1.进入国际排行榜)
ps.p_choice = raw_input()
ps.Program_Start()

来看一下运行的结果吧（今天刚好前三名分别是多图，单图，和动图）