爬虫文章（xpath+正则）

推荐一本书《为何爱会伤人》

读了真的受益匪浅

这里加个3分钟读一本书的链接  https://v.douyin.com/i8F4HnGK/

找个阅读网站

爬取效果

完整代码

只用到requests,lxml库

每个章节保存为一个txt

​
import requests,re,os
from lxml import etree
dv=os.path.join(os.path.expanduser("~"),"Desktop\读书")
if not os.path.exists(dv):
    os.mkdir(dv)
url='https://www.wxzpyd.com/novel/catalog/610fadf97dbb3f28a8755413_1?cur=2&page=1&source=%2Fnovel%2Fchapter%2F209_610fadf97dbb3f28a8755413_610fadfc7dbb3f28a8755421.html%3Fpage%3D1%26pagecount%3D1%26tt%3D7758258'
gu=requests.get(url)
gul=etree.HTML(gu.text)
gus=gul.xpath('//*[@id="app"]/div[2]/main/ul/li')
a=0   #第几章节
for i in gus:
    name = i.xpath('a/text()')[0]
    try:
        na=name.split(' ')[2]
        me=name.split(' ')[3]
        name=na+me
    except:
        name=name
    sr = i.xpath('a/@href')[0]
    src = 'https://www.wxzpyd.com/' + sr
    gumo=requests.get(src)
    pattern = re.compile(r'
(.*?)
')  # 匹配 
 标签内的文本内容
    # 使用正则表达式查找匹配项
    matches = pattern.findall(gumo.text)
    with open(dv+'/'+str(a)+name+'.txt', 'w', encoding='utf-8') as file:
        # 将匹配到的内容逐行写入文件
        for match in matches:
            file.write(match + '\n')
    a=a+1
    print(name+'-下载完成')

​

所有章节保存为一个txt

import requests,re,os
from lxml import etree
dv=os.path.join(os.path.expanduser("~"),"Desktop\读书")
if not os.path.exists(dv):
    os.mkdir(dv)
url='https://www.wxzpyd.com/novel/catalog/610fadf97dbb3f28a8755413_1?cur=2&page=1&source=%2Fnovel%2Fchapter%2F209_610fadf97dbb3f28a8755413_610fadfc7dbb3f28a8755421.html%3Fpage%3D1%26pagecount%3D1%26tt%3D7758258'
gu=requests.get(url)
gul=etree.HTML(gu.text)
gus=gul.xpath('//*[@id="app"]/div[2]/main/ul/li')
a=0   #第几章节
with open(dv+'/'+'为何爱会伤人'+'.txt', 'w', encoding='utf-8') as file:
    for i in gus:
        sr = i.xpath('a/@href')[0]
        src = 'https://www.wxzpyd.com/' + sr
        gumo=requests.get(src)
        pattern = re.compile(r'
(.*?)
')  # 匹配 
 标签内的文本内容
        # 使用正则表达式查找匹配项
        matches = pattern.findall(gumo.text)
            # 将匹配到的内容逐行写入文件
        for match in matches:
            file.write(match + '\n')

保存后可转到微信读书导入书架（该书为付费书籍）

可以转存为word

from docx import Document
# 创建一个新的 Word 文档
doc = Document()
# 将匹配到的内容写入 Word 文档
...
for match in matches:
    doc.add_paragraph(match)
# 保存 Word 文档
doc.save('matched_content.docx')

仅用来练手 无其他用途