【Python爬虫】-03- 阳光电影的下载链接爬取

作业分析

爬取阳光电源网站的下载链接

按照指导老师说的要分为三步走

第一步：确定爬取的内容（电影页的前11个分类）
第二步：分析页面的结构，爬取页面数（正则）和电影的分类标签，以此构造分页url，然后爬取电影的名字和url
第三步：最后在详细页爬取电影的下载地址。

代码：

import requests
import re
from lxml import etree

def get_cate_info(url):#创建函数
    res = requests.get(url)
    res.encoding = 'gb2312'#enconding表示的是页面接受的数据编码类型
    html = etree.HTML(res.text)
    infos = html.xpath('//div[@class="contain"]/ul/li[position()<12]')
    for info in infos:
        cate_name = info.xpath('a/text()')[0]
        cate_url = res.url + info.xpath('a/@href')[0]
        get_movie(cate_url,cate_name)

def get_movie(url,cate_name):
    res = requests.get(url)
    res.encoding = 'gb2312'
    all_page = re.findall('共(.*?)页', res.text)#计算共需要爬取多少页
    kind = re.findall(' 0:
        kind_url = url.rstrip(url.split('/')[-1]) + str(kind[0])
        for page in range(1,int(all_page[0])+1):
            page_url = kind_url + str(page) + '.html'
            resp = requests.get(page_url)
            resp.encoding = 'gb2312'
            html = etree.HTML(resp.text)
            infos = html.xpath('//div[@class="co_content8"]/ul//table')
            for info in infos:
                detail_url = 'http://www.ygdy8.com' + info.xpath('tr[2]/td[2]/b/a/@href')[0]
                movie_name = info.xpath('tr[2]/td[2]/b/a/text()')[0]
                print(detail_url)
                get_resource(detail_url, cate_name, url, movie_name)

def get_resource(url,cate_name,cate_url,movie_name):
    res = requests.get(url)
    res.encoding = 'gb2312'
    html = etree.HTML(res.text)
    movie_resource = html.xpath('//tbody//tr/td/a/text()')[0]
    print(movie_resource)

if __name__ == '__main__':
    urls = ['http://www.ygdy8.com'.format(str(i)) for i in range(1,94)]
    for url in urls:
        get_cate_info(url)

运行结果：

"C:\Program Files\Python36\python.exe" "D:/小克学习/python/项目/week four/sunny-movie1.py"
http://www.ygdy8.com/html/gndy/dyzz/20170806/54699.html
ftp://ygdy8:[email protected]:7025/[阳光电影www.ygdy8.com].巴霍巴利王(下)：终结.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170802/54644.html
ftp://ygdy8:[email protected]:8021/[阳光电影www.ygdy8.com].异形：契约.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170802/54643.html
ftp://ygdy8:[email protected]:7022/[阳光电影www.ygdy8.com].52赫兹，我爱你.BD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170801/54638.html
ftp://ygdy8:[email protected]:7021/[阳光电影www.ygdy8.com].一锤定音.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170801/54637.html
ftp://ygdy8:[email protected]:8031/[阳光电影www.ygdy8.com].敌对区域.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170731/54632.html
ftp://ygdy8:[email protected]:7020/[阳光电影www.ygdy8.com].逆时营救.HD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170731/54631.html
ftp://ygdy8:[email protected]:8020/[阳光电影www.ygdy8.com].师父.BD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170728/54613.html
ftp://ygdy8:[email protected]:7018/[阳光电影www.ygdy8.com].目击者之追凶.HD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170728/54612.html
ftp://ygdy8:[email protected]:8028/[阳光电影www.ygdy8.com].我心雀跃.HD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170727/54606.html
ftp://ygdy8:[email protected]:7017/[阳光电影www.ygdy8.com].亚瑟王：斗兽争霸.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170727/54605.html
ftp://ygdy8:[email protected]:8017/[阳光电影www.ygdy8.com].指甲刀人魔.BD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170726/54600.html
ftp://ygdy8:[email protected]:8016/[阳光电影www.ygdy8.com].你的名字。.BD.720p.日语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170726/54595.html
ftp://ygdy8:[email protected]:7016/[阳光电影www.ygdy8.com].奥斯曼中尉.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170725/54594.html
ftp://ygdy8:[email protected]:8015/[无极电影-www.wujidy.com].记忆大师.HD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170725/54593.html
ftp://ygdy8:[email protected]:7015/[无极电影-www.wujidy.com].无罪之城.HD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170724/54588.html
ftp://ygdy8:[email protected]:8014/[阳光电影www.ygdy8.com].三个老枪手.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170724/54587.html
ftp://ygdy8:[email protected]:7013/[阳光电影www.ygdy8.com].首杀.HD.720p.中英双字幕.rmvb
http://www.ygdy8.com/html/gndy/dyzz/20170723/54575.html
ftp://ygdy8:[email protected]:7012/[阳光电影www.ygdy8.com].一念无明.HD.720p.粤语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170723/54574.html
ftp://ygdy8:[email protected]:8012/[阳光电影www.ygdy8.com].共助.BD.720p.韩语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170722/54573.html
ftp://ygdy8:[email protected]:8026/[阳光电影www.ygdy8.com].台北物语.HD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170721/54566.html
ftp://ygdy8:[email protected]:8025/[阳光电影www.ygdy8.com].忠爱无言.HD.720p.国语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170719/54549.html
ftp://ygdy8:[email protected]:8024/[阳光电影www.ygdy8.com].我们停战吧.BD.720p.国粤双语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170713/54504.html
ftp://ygdy8:[email protected]:8022/[阳光电影www.ygdy8.com].攻壳机动队.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170712/54497.html
ftp://ygdy8:[email protected]:8003/[阳光电影www.ygdy8.com].天才少女.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170712/54496.html
ftp://ygdy8:[email protected]:7047/[阳光电影www.ygdy8.com].走火交易.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170710/54483.html
ftp://ygdy8:[email protected]:8045/[阳光电影www.ygdy8.com].宝贝老板.BD.720p.国粤英三语.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170707/54458.html
ftp://ygdy8:[email protected]:8044/[阳光电影www.ygdy8.com].毒。诫.BD.720p.国粤双语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170707/54457.html
ftp://ygdy8:[email protected]:8020/[阳光电影www.ygdy8.com].攻壳机动队.HD.720p.国英双语.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170706/54451.html
ftp://ygdy8:[email protected]:7044/[阳光电影www.ygdy8.com].承诺.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170705/54441.html
ftp://ygdy8:[email protected]:8043/[阳光电影www.ygdy8.com].迷失Z城.BD.720p.中英双字幕.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170705/54440.html
ftp://ygdy8:[email protected]:8019/[阳光电影www.ygdy8.com].将来的事.BD.720p.法语中字.mkv
http://www.ygdy8.com/html/gndy/dyzz/20170704/54433.html

Process finished with exit code 1

老师的代码是写入到csv中，我把这部分改掉了。