【爬虫】豆瓣电影爬取

爬虫的原理：

1. 模拟浏览器的行为，通过网络请求将目标网页抓取到本地。
2. 使用一定的匹配规则，将目标网页中需要的数据提取出来，把不需要的过滤掉。
3. 根据需求，把提取出来的数据存储到磁盘中（json、csv、excel、数据库）。

需要安装的库：

1. requests：用来做网络请求的。就跟浏览器是一样的。安装方式：

   pip install requests
   

2. bs4：用来将请求下来的数据进行解析的。安装方式：

   pip install bs4
   

3. lxml：这个库是用来解析html和xml格式数据的。BeautifulSoup相当于只是一个壳，底层还是要基于lxml类似的这种解析器来解析。html5lib、html.parse。也要安装。安装方式如下：

   pip install lxml
   

#encoding: utf-8

import requests
from bs4 import BeautifulSoup
import json

def get_page():
    # 1. url
    url = "https://movie.douban.com/cinema/nowplaying/changsha/"
    # 2. 请求页面的时候应该发送什么数据
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"
    }
    # 3. GET/POST请求
    # 采用的是GET请求

    # 4. 发送请求
    response = requests.get(url, headers=headers)
    text = response.text
    return text

def parse_page(text):
    soup = BeautifulSoup(text,'lxml')
    movies = []
    # data-category=nowplaying
    # attribute
    liList = soup.find_all("li",attrs={"data-category":"nowplaying"})
    for li in liList:
        movie = {}
        title = li['data-title']
        score = li['data-score']
        release = li['data-release']
        duration = li['data-duration']
        region = li['data-region']
        director = li['data-director']
        actors = li['data-actors']
        img = li.find('img')
        thumbnail = img['src']

        movie['title'] = title
        movie['score'] = score
        movie['release'] = release
        movie['duration'] = duration
        movie['region'] = region
        movie['director'] = director
        movie['actors'] = actors
        movie['thumbnail'] = thumbnail
        movies.append(movie)
    return movies

def save_data(data):
    with open('douban.json','w',encoding='utf-8') as fp:
        # json.dump作用：
        # 将字典、列表dump成满足json格式的字符串
        json.dump(data,fp,ensure_ascii=False)


if __name__ == '__main__':
    text = get_page()
    movies = parse_page(text)
    save_data(movies)