爬虫－scrawler

python爬虫常用库

requests/urllib/urllib2 执行http请求，url/base64编码解码等操作
selenium 自动化浏览器交互，xpath定位元素，动态执行js
lxml 用于解析html，xpath定位元素
BeautifulSoup 封装和简化了xpath定位元素

xpath语法

http://www.w3school.com.cn/xpath/xpath_syntax.asp
以https://www.baidu.com为例

基本语法
/                   子节点
//                  根节点
nodename            节点                   //input
@attrib             属性                   //input[@type]
.                   当前节点                //input/.         //input[@type='submit']/.
..                  父节点                 //input/..        //input[@type='submit']/..
[]                  匹配元素下标            //input[1]        //input[@type='submit'][1]
last()              最后一个下标            //input[last()]   //input[last()-1]  
position()          当前下标               //input[position()<3]
*                   任何元素               //form[@id='form']/*
@*                  任何有属性的元素        //form[@*] 

运算符
||                  多个xpath路径          //form | //input              
+ - * div mod       数学计算
> = < != >= <=      条件表达式             //input[@type='hidden'][@value=8]
or and              逻辑表达式             //input[@name='ie' or @name='f']

python发送http请求

以urllib为例
python2 import urllib as urllib_
python3 import urllib.request as urllib_

设置代理

opener = urllib_.build_opener(urllib_.ProxyHandler({
    "http": proxy,
    "https": proxy,
}))
response = opener.open(urllib_.Request(url)).read()

设置header

headers = {
    'User-Agent': 'Mozilla/5.0'
}
opener = urllib_.build_opener()
response = opener.open(urllib_.Request(url, headers=headers)).read()

GET请求

opener = urllib_.build_opener()
response = opener.open(urllib_.Request(url)).read()

POST请求

postata = 'abc'
opener = urllib_.build_opener()
response = opener.open(urllib_.Request(url, data=postdata)).read()

selenium

selenium支持IE／Firefox／Chrome，在配置python+selenium环境时首先pip安装selenium，同时下载合适的内核，如chrome为http://chromedriver.storage.googleapis.com/index.html。不知道下载哪个内核可以看运行时报错信息

学习步骤

最简单的爬虫的核心代码

#! /usr/bin/env python
# # -*- coding: utf-8 -*-

import datetime
import hashlib
import json
import logging
from lxml import etree
import os
import random
import re
import socket
import ssl
import sys
import threading
import threadpool
import time


def RequestWithDefProxy(url, headers, postdata, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
    try:
        opener = urllib_.build_opener()
        content = opener.open(urllib_.Request(url, headers=headers, data=postdata), timeout=timeout).read()
        return content
    except Exception as e:
        print(url, e)
        return None
    
response = RequestWithDefProxy('http://tu.duowan.com/tag/27812.html', dict(), None)
html = etree.HTML(response)
items = html.xpath('//li[@class="box"]')
for item in items:
    suburl = item.xpath('a/img/@src')[0]
    filename = os.path.basename(suburl)
    subresponse = RequestWithDefProxy(suburl, "", None)
    if subresponse is not None:
        with open(filename, 'wb') as img:
            img.write(subresponse)

要点：

实现get请求和post请求
使用xpath在html中定位元素
将RequestWithDefProxy得到的数据保存为html然后分析xpath

....

selenium使用

安装python+selenium

安装python接口：pip install -i https://pypi.tuna.tsinghua.edu.cn/simple selenium

选择浏览器并安装driver

selenium -> driver -> chrome

Firefox Chrome Ie Edge Opera Safari 
headless: 
   Firefox   geckodriver
   Chrome    chromedriver

http://chromedriver.storage.googleapis.com/index.html
https://github.com/mozilla/geckodriver/releases

headless

# headless chrome
from selenium import webdriver
options = webdriver.chrome.options.Options()
options.add_argument('--headless')
driver = webdriver.Chrome(chrome_options=options)
driver.get("http://www.baidu.com")
baiduyixia = driver.find_element_by_xpath("//input[@type='submit']")
keyword = driver.find_element_by_xpath("//input[@id='kw']")
print(baiduyixia, keyword)

# headless firefox
from selenium import webdriver
options = webdriver.firefox.options.Options()
options.add_argument('--headless')
driver = webdriver.Firefox(firefox_options=options)
driver.get("http://www.baidu.com")
baiduyixia = driver.find_element_by_xpath("//input[@type='submit']")
keyword = driver.find_element_by_xpath("//input[@id='kw']")
print(baiduyixia, keyword)

4.元素选取

find_element_*     find_elements_*
id定位：find_element_by_id()
name定位：find_element_by_name()
class定位：find_element_by_class_name()
link定位：find_element_by_link_text()
partial link定位：find_element_by_partial_link_text()
tag定位：find_element_by_tag_name()
xpath定位：find_element_by_xpath()
css定位：find_element_by_css_selector()

browser.get("http://www.baidu.com")
#########百度输入框的定位方式##########
#通过id方式定位
browser.find_element_by_id("kw").send_keys("selenium")
#通过name方式定位
browser.find_element_by_name("wd").send_keys("selenium")
#通过tag name方式定位
browser.find_element_by_tag_name("input").send_keys("selenium")
#通过class name方式定位
browser.find_element_by_class_name("s_ipt").send_keys("selenium")
#通过CSS方式定位
browser.find_element_by_css_selector("#kw").send_keys("selenium")
#通过xpath方式定位
browser.find_element_by_xpath("//input[@id='kw']").send_keys("selenium")
############################################
browser.find_element_by_id("su").click()


新闻
hao123
通过link text定位:
dr.find_element_by_link_text("新闻")
dr.find_element_by_link_text("hao123")
通过link text定位:
dr.find_element_by_partial_link_text("新")
dr.find_element_by_partial_link_text("hao")
dr.find_element_by_partial_link_text("123")

操作

implicitly_wait：等待
current_url：获取当前url
click：点击
send_keys：模拟按键   
clear：文本置空  
text：获取文本
title：
page_source：渲染后的html
id：获取id
execute_script：执行js
execute_async_script
get_attribute：获取属性
get_screenshot_as_base64  元素也可以截图
get_screenshot_as_file
get_screenshot_as_png

search_text = driver.find_element_by_id('kw')
search_text.send_keys('selenium')
search_text.submit()

driver.find_element_by_id("user_name").send_keys(Keys.TAB) 
time.sleep(3) 
driver.find_element_by_id("user_pwd").send_keys("123456")
driver.find_element_by_id("user_pwd").send_keys(Keys.ENTER)
driver.find_element_by_id("kw").send_keys(Keys.SPACE)
driver.find_element_by_id("kw").send_keys(Keys.BACK_SPACE)
driver.find_element_by_id("login").send_keys(Keys.ENTER) 
driver.find_element_by_id("kw").send_keys(Keys.CONTROL,'a')
driver.find_element_by_id("kw").send_keys(Keys.CONTROL,'x')

click() 单击
context_click()  右击；
move_to_element()  鼠标悬停。

driver.get("https://www.baidu.cn")
# 定位到要悬停的元素
above = driver.find_element_by_link_text("设置")
# 对定位到的元素执行鼠标悬停操作
ActionChains(driver).move_to_element(above).perform()

frame切换

import time
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException

browser = webdriver.Chrome()
url = 'http://www.runoob.com/try/try.php?filename=jqueryui-api-droppable'
browser.get(url)
browser.switch_to.frame('iframeResult')
source = browser.find_element_by_css_selector('#draggable')
print(source)
try:
    logo = browser.find_element_by_class_name('logo')
except NoSuchElementException:
    print('NO LOGO')
browser.switch_to.parent_frame()
logo = browser.find_element_by_class_name('logo')
print(logo.text)

driver.get("http://www.126.com")
driver.switch_to.frame('x-URS-iframe')
driver.find_element_by_name("email").clear()
driver.find_element_by_name("email").send_keys("username")
driver.find_element_by_name("password").clear()
driver.find_element_by_name("password").send_keys("password")
driver.find_element_by_id("dologin").click()
driver.switch_to.default_content()

windows切换

driver.get("http://www.baidu.com")
# 获得百度搜索窗口句柄
sreach_windows = driver.current_window_handle
driver.find_element_by_link_text('登录').click()
driver.find_element_by_link_text("立即注册").click()
# 获得当前所有打开的窗口的句柄
all_handles = driver.window_handles
# 进入注册窗口
for handle in all_handles:
    if handle != sreach_windows:
        driver.switch_to.window(handle)
        print('now register window!')
        driver.find_element_by_name("account").send_keys('username')
        driver.find_element_by_name('password').send_keys('password')
        time.sleep(2)
        # ……

警告框

from selenium.webdriver.common.action_chains import ActionChains
driver.get('http://www.baidu.com')
# 鼠标悬停至“设置”链接
link = driver.find_element_by_link_text('设置')
ActionChains(driver).move_to_element(link).perform()
# 打开搜索设置
driver.find_element_by_link_text("搜索设置").click()
# 保存设置
driver.find_element_by_class_name("prefpanelgo").click()
time.sleep(2)
# 接受警告框
driver.switch_to.alert.accept()

9.下拉框选择

from selenium.webdriver.support.select import Select
driver.get('http://www.baidu.com')
# 鼠标悬停至“设置”链接
driver.find_element_by_link_text('设置').click()
sleep(1)
# 打开搜索设置
driver.find_element_by_link_text("搜索设置").click()
sleep(2)
# 搜索结果显示条数
sel = driver.find_element_by_xpath("//select[@id='nr']")
Select(sel).select_by_value('50')  # 显示50条

cookie操作

add_cookie
get_cookies
delete_all_cookies
delete_cookie
driver.get("http://www.youdao.com")
# 获得cookie信息
cookie= driver.get_cookies()
# 将获得cookie的信息打印
print(cookie)

driver.get("http://www.youdao.com")
# 向cookie的name 和value中添加会话信息
driver.add_cookie({'name': 'key-aaaaaaa', 'value': 'value-bbbbbb'})
# 遍历cookies中的name 和value信息并打印，当然还有上面添加的信息
for cookie in driver.get_cookies():
    print("%s -> %s" % (cookie['name'], cookie['value']))

js调用

driver.get("http://www.baidu.com")
# 设置浏览器窗口大小
driver.set_window_size(500, 500)
# 搜索
driver.find_element_by_id("kw").send_keys("selenium")
driver.find_element_by_id("su").click()
sleep(2)
# 通过javascript设置浏览器窗口的滚动条位置
js="window.scrollTo(100,450);"
driver.execute_script(js)
sleep(3)


输入文本：
# 第一种方式：模拟键盘信息
keyword.send_keys("测试一下")
baiduyixia.click()
# 第二种方式：执行js
driver.execute_script('document.getElementById("kw").value="bbb"')

Python爬虫笔记

爬虫－scrawler

分类

python爬虫常用库

xpath语法

python发送http请求

设置代理

设置header

GET请求

POST请求

selenium

学习步骤

最简单的爬虫的核心代码

....

....

....

....

....

selenium使用

安装python+selenium

选择浏览器并安装driver

headless

你可能感兴趣的:(Python爬虫笔记)