【爬虫】用wget命令爬虫的简易教程
文章目录
- 1. 获取登录的请求
- 2. 用postman模拟登录请求
- 3. 用wget模拟登录请求并保存cookie
- 4. 开始爬取网站
- 5. 查看爬取结果
- 6. 网站爬虫简易教程
爬取需要登录的网站的资源
背景：对于一些网站需要使用用户名和密码登录并且使用了https，我们如果不通过凭证将无法进行该网站的下载、爬虫！，而具体的凭证一般的是”cookies“形式的。
内容：本文主要介绍了如何爬取需要登录网站的内容（视频、图片、网页）的简易教程。
postman文档地址：https://learning.postman.com/docs/sending-requests/requests/
1. 获取登录的请求
首先需要使用用户名密码登录到网站，查看f12找到登录的请求，复制成Copy as CURL
登录请求uri一般是login或register等等，认真找一找
2. 用postman模拟登录请求
- 导入请求到postman
将复制的内容导入到postman接口工具中
- 发送请求，获取到wget代码片段
发送请求，检查是否模拟登录成功，如果请求发送成功，则按下图获取到postman的wget代码片段。
3. 用wget模拟登录请求并保存cookie
- 在从postman复制的代码片段后追加（如下）cookie配置。
意思就是把cookie保存在cookies.txt中，以及后续使用
--save-cookies=cookies.txt --keep-session-cookies
- 模拟登录请求并保存cookie
用命令行发送类似下面的wget命令。该命令就是postman复制的代码片段后追加--save-cookies=cookies.txt --keep-session-cookies
wget --no-check-certificate --quiet --method GET --timeout=0 --header 'authority: qvb111.xyz' --header 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' --header 'accept-language: zh-CN,zh;q=0.9' --header 'cache-control: max-age=0' --header 'cookie: md10=kdfjijf89485.online; _ga=GA1.1.1107869110.1654255726; _ga_6DLS4FBHC6=GS1.1.1654259056.2.1.1654260355.0; _nipple_session=DZmMES3vGmHhXLnp9TnULezhbUhy%2FIqFyLNWNYot0S%2FCq7n73iJ1P7ypivBy4u8IPPYe6smeiP7I%2FttFSLEHeb6jEafg50to7ceYCtDLQdAVwnBRdGenEKtc7dODRRQn9FaVOS9ietmoMO0IAbcJ6%2B%2BypZestlQ9IIoAYyYmTvmzQltULHnuA2cQEGUyxlmJqwCF1nfYrhMtBqEgpFP2UwrBKEcBBcqYFL96klIQBOOCSdm8UueNKLZ9O%2BUAlN%2FEIRQgV229ziwy5kUVxBDYzJ9tmLbxrVtSKzKxESuQ1W9n6JefP64fB%2FC7l7kWfL0Vys%2BlCi57UkpuhHfM0IJhj33FOSy4iMtXcVGETor4NG2%2FHcUL2U974YCfPBX6Rc%2BoQ%2Bm8%2Fkyzdutme9AQS%2FPk--RkCe6gHEAt3X3JgH--j5UScZwkeVHIukpKpt6TGQ%3D%3D; _nipple_session=GBgJoGvRuRJBkWfWwcoSDKiquxucPgj24AUTQQe%2FfPANRvWA6unhiGQFQ8SPqml271vlZwFtGra448GmgDKSnpX%2FCSUkwzEiqDr0ekV9oKw%2FKdrkk6ELO0Z3J8YqInUSiQKm04eVKJvHCRc5p0MH1jJ%2BZAcONVfvfh11Ai2TGpTzYOxZ%2BIi2uHqXn817GUFO7GkDB2VI%2FTIPMz%2B8J7Sxj2GJaEQU%2FKyROs5XN0BWCVhe9EF8CT8RKa1DP%2FrLzOosn33weZOCaPR%2Bbn7jwupxrxsCZ68Tg9oUl%2Ff4GrVTPoAyaWuoPlD0sKtteh9HKqg%2Fb%2BzJMS04US9OlztCm5rzJmV7xW6uoUX9%2BerYxZJB11haN%2Fquablym5VufyWURAZybjY7jEaCoSp94t4EBlPJ--SphXN3nrbR%2Fc3Yhu--G6JqS5oBVQSPdSCeXCf4lg%3D%3D' --header 'referer: https://qvb111.xyz/users/sign_in' --header 'sec-ch-ua: "-Not.A/Brand";v="8", "Chromium";v="102"' --header 'sec-ch-ua-mobile: ?0' --header 'sec-ch-ua-platform: "macOS"' --header 'sec-fetch-dest: document' --header 'sec-fetch-mode: navigate' --header 'sec-fetch-site: same-origin' --header 'sec-fetch-user: ?1' --header 'upgrade-insecure-requests: 1' --header 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36' 'https://qvb111.xyz/' --save-cookies=cookies.txt --keep-session-cookies
4. 开始爬取网站
配置从cookies.txt中加载cookies，并爬取网站https://qvb111.xyz/girl/show/2797
wget --load-cookies cookies.txt \
--keep-session-cookies \
https://qvb111.xyz/girl/show/2797
5. 查看爬取结果
作者爬取了某个带颜色的网站后，并用以下的命令查看爬取的内容
cd firefish
ls
cd show
ls
ls | wc -l
du -sh .
6. 网站爬虫简易教程
1、正常登录目标网站
2、找到登录请求、复制、导入postman处理
3、复制postman生成wget代码片段，并追加设置
--save-cookies cookies.txt --keep-session-cookies
4、模拟登录并保存凭证
wget --no-check-certificate --quiet --method GET --timeout=0 --header 'authority: qvb111.xyz' --header 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' --header 'accept-language: zh-CN,zh;q=0.9' --header 'cache-control: max-age=0' --header 'cookie: md10=kdfjijf89485.online; _ga=GA1.1.1107869110.1654255726; _ga_6DLS4FBHC6=GS1.1.1654259056.2.1.1654260355.0; _nipple_session=DZmMES3vGmHhXLnp9TnULezhbUhy%2FIqFyLNWNYot0S%2FCq7n73iJ1P7ypivBy4u8IPPYe6smeiP7I%2FttFSLEHeb6jEafg50to7ceYCtDLQdAVwnBRdGenEKtc7dODRRQn9FaVOS9ietmoMO0IAbcJ6%2B%2BypZestlQ9IIoAYyYmTvmzQltULHnuA2cQEGUyxlmJqwCF1nfYrhMtBqEgpFP2UwrBKEcBBcqYFL96klIQBOOCSdm8UueNKLZ9O%2BUAlN%2FEIRQgV229ziwy5kUVxBDYzJ9tmLbxrVtSKzKxESuQ1W9n6JefP64fB%2FC7l7kWfL0Vys%2BlCi57UkpuhHfM0IJhj33FOSy4iMtXcVGETor4NG2%2FHcUL2U974YCfPBX6Rc%2BoQ%2Bm8%2Fkyzdutme9AQS%2FPk--RkCe6gHEAt3X3JgH--j5UScZwkeVHIukpKpt6TGQ%3D%3D; _nipple_session=GBgJoGvRuRJBkWfWwcoSDKiquxucPgj24AUTQQe%2FfPANRvWA6unhiGQFQ8SPqml271vlZwFtGra448GmgDKSnpX%2FCSUkwzEiqDr0ekV9oKw%2FKdrkk6ELO0Z3J8YqInUSiQKm04eVKJvHCRc5p0MH1jJ%2BZAcONVfvfh11Ai2TGpTzYOxZ%2BIi2uHqXn817GUFO7GkDB2VI%2FTIPMz%2B8J7Sxj2GJaEQU%2FKyROs5XN0BWCVhe9EF8CT8RKa1DP%2FrLzOosn33weZOCaPR%2Bbn7jwupxrxsCZ68Tg9oUl%2Ff4GrVTPoAyaWuoPlD0sKtteh9HKqg%2Fb%2BzJMS04US9OlztCm5rzJmV7xW6uoUX9%2BerYxZJB11haN%2Fquablym5VufyWURAZybjY7jEaCoSp94t4EBlPJ--SphXN3nrbR%2Fc3Yhu--G6JqS5oBVQSPdSCeXCf4lg%3D%3D' --header 'referer: https://qvb111.xyz/users/sign_in' --header 'sec-ch-ua: "-Not.A/Brand";v="8", "Chromium";v="102"' --header 'sec-ch-ua-mobile: ?0' --header 'sec-ch-ua-platform: "macOS"' --header 'sec-fetch-dest: document' --header 'sec-fetch-mode: navigate' --header 'sec-fetch-site: same-origin' --header 'sec-fetch-user: ?1' --header 'upgrade-insecure-requests: 1' --header 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36' 'https://qvb111.xyz/' --save-cookies=cookies.txt --keep-session-cookies
5、开始爬虫
wget --load-cookies cookies.txt \
--keep-session-cookies \
https://qvb111.xyz/girl/show/2797
6、查看爬虫成果（见视频）
可以以个人网站测试或gitee个人仓库测试，不合理使用