爬虫与信息提取 1-1 - Requests库入门

你是无意穿堂风偏偏孤倨引山洪

1进度

安装 requests

conda install requests

http://www.python-requests.org/en/master/

# -*- coding: utf-8 -*-
import requests
r = requests.get("http://www.mi.com")
print(r.status_code)
r.encoding = 'utf-8'
print(r.text)

requests 库7个主要方法：

requests 库7个主要方法

requests - get()

r = requests.get(url)

r = requests.get("http://www.mi.com")

get()1

get()2

get()3

requests库

response 对象的属性：

response 对象的属性

status_code

理解response的编码：

理解response的编码

爬取网页的通用代码框架：

网络连接有风险
异常处理很重要

理解Requests库的异常：

Requests库的异常1

Requests库的异常2

# -*- coding: utf-8 -*-

import requests

def getHTMLText(url):
    try:
        r = requests.get(url, timeout=30)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        return r.text
    except:
        return "产生异常"

if __name__ == "__main__":
    url = "http://www.mi.com"
    print(getHTMLText(url))

HTTP协议及Requests库方法：

Http协议：

Http协议

url

url

Http协议对资源的操作：

Http协议对资源的操作1

Http协议对资源的操作2

http协议通过url对资源定位通过以上6个操作方法对资源进行管理每次操作都是独立无状态的（两次操作间没有关系）

区别

Requests库的head()：

# -*- coding: utf-8 -*-
import requests
r = requests.get("http://www.mi.com")
print(r.status_code)
r.encoding = 'utf-8'
print(r.headers)

Requests库的post()：

post()

# -*- coding: utf-8 -*-

import requests

payload = {'key1':'value1', 'key2':'value2'}

r = requests.post('http://httpbin.org/post', data=payload)

print(r.text)

print:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "key1": "value1", 
    "key2": "value2"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "close", 
    "Content-Length": "23", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.14.2"
  }, 
  "json": null, 
  "origin": "43.247.4.53", 
  "url": "http://httpbin.org/post"
}