import requests
from bs4 import BeautifulSoup as bf
from lxml import etree
url ='https://www.soxscc.com/MangHuangJi/'
headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'}
resp = requests.get(url,headers=headers)
resp_xpath = etree.HTML(resp.text)
result = resp_xpath.xpath('//*[@id="novel4451"]/dl/dd/a/@href')
for i in range(0,4):
url1 ='https://www.soxscc.com/' +result[i]
print(url1)
输出结果是
因为设置只循环四次，从第一章节循环到第四章网址
for i in range(0,4):
url1 ='https://www.soxscc.com/' +result[i]
print(url1)
用xpath方便了很多，这次只需要在网页检查的页面点击复制xpath
之后
result = resp_xpath.xpath('//*[@id="novel4451"]/dl/dd/a/@href')
就得到了href列表result,添加一些代码打印出来
for i in result:
print()
方便多了。。。
之后就更简单了，只用了3次for循环，而且不复杂
import requests
from bs4import BeautifulSoupas bf
from lxmlimport etree
url ='https://www.soxscc.com/MangHuangJi/'
headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'}
resp = requests.get(url,headers=headers)
resp_xpath = etree.HTML(resp.text)
result = resp_xpath.xpath('//*[@id="novel4451"]/dl/dd/a/@href')
for i in range(0,10):
output ="\n\n{}\n\n{}\n\n\n\n\n\n"
url1 ='https://www.soxscc.com/' +result[i]
resp1 = requests.get(url1,headers=headers)
soup = bf(resp1.text,'lxml')
title = soup.find('h1').string
contents = soup.findAll('div',class_='content')
for i in contents:
content = i.get_text()
output1 = output.format(title,content)
for i in output1:
with open('souxiaoshuo.txt','a',encoding='utf-8')as f:
f.write(i)
这次可以指定下载多少章节，10次循环下载10章节
下次爬取网抑云音乐评论。。。
假如获得了网页的字符串格式，也就是上面的resp.text,用etree可以把它变成xpath的格式，也就是说可以用.xpath()这个函数，括号里面写要爬取的东西的xpath,直接在网页源代码里复制就行了。有时候复制了也爬取不了。
/@href就是取出标签a里的那一段地址
如果是/text()的话，会出现文字信息。
.format()的作用就是把括号里的东西填入{}中去
比如
得到