目标网站 https://divnil.com
首先看看这网站是怎样加载数据的;
打开网站后发现底部有下一页的按钮，ok，爬这个网站就很简单了;
我们目标是获取每张图片的高清的源地址，并且下载图片到桌面;
先随便打开一张图片看看详细;
emmm，只有一张图
看起来还挺清晰的，单击新窗口打开图片
然后下载图片，说实话，这图片很小，我很担心不是高清原图(管他的);
PS:一定要禁用广告拦截插件，不然加载不出图，我就在这被坑T_T;
接着分析我们从何入手
1、先去主页面获取每个图片的详细页面的链接
这链接还是比较好获取的，直接 F12 审核元素，或者右键查看代码，手机上chrome和firefox在url前面加上 "view-source"
比如：view-source:https://www.baidu.com/
2、从详细页面获取图片大图地址
随便打开一个图片详细页面如图：
接着按 F12 审核元素，我们需要定位该图片的链接，首先单击左上角的这玩意儿，像一个鼠标的图标：
接着只需要单击网页上的图片就能定位到代码了：
3、用大图地址下载该图片
这个很简单，看代码
先安装 Requests 和 BeautifulSoup 库
pipinstallrequests bs4
导入库
importrequestsfrom bs4importBeautifulSoupimport sys
请求获取网页源代码
url ="https://divnil.com/wallpaper/iphone8/%E3%82%A2%E3%83%8B%E3%83%A1%E3%81%AE%E5%A3%81%E7%B4%99_2.html"
headers = {
"User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0",
}
resp = requests.get(url, headers=headers)
ifresp.status_code != requests.codes.OK:
print("Request Error, Code: %d"% resp.status_code)
sys.exit()
然后解析出所有图片的详细地址
soup = BeautifulSoup(resp.text,"html.parser")
contents = soup.findAll("div", id="contents")[0]
wallpapers = contents.findAll("a", rel="wallpaper")
links = []
forwallpaper in wallpapers:
links.append(wallpaper['href'])
接着在详细网页里获取那个看似高清的图片的不确定是否为真实图片链接并下载(/滑稽)
import os
head ="https://divnil.com/wallpaper/iphone8/"
ifos.path.exists("./Divnil") != True:
os.mkdir("./Divnil")
forurl in links:
url = head + url
resp = requests.get(url, headers=headers)
ifresp.status_code != requests.codes.OK:
print("URL: %s REQUESTS ERROR. CODE: %d"% (url, resp.status_code))
continue
soup = BeautifulSoup(resp.text,"html.parser")
img = soup.find("div", id="contents").contents.find("img", id="main_content")
img_url = head + img['"original'].replace("../","")
img_name = img['alt']
print("start download %s ..."% img_url)
resp = requests.get(img_url, headers=headers)
ifresp.status_code != requests.codes.OK:
print("IMAGE %s DOWNLOAD FAILED."% img_name)
withopen("./Divnil/"+ img_name +".jpg","wb")asf:
f.write(resp.content)
主要代码
完成，贴上所有代码
importrequests
frombs4importBeautifulSoup
importsys
importos
classDivnil:
def__init__(self):
self.url ="https://divnil.com/wallpaper/iphone8/%E3%82%A2%E3%83%8B%E3%83%A1%E3%81%AE%E5%A3%81%E7%B4%99.html"
self.head ="https://divnil.com/wallpaper/iphone8/"
self.headers = {
"User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0",
}
defgetImageInfoUrl(self):
resp = requests.get(self.url, headers=self.headers)
ifresp.status_code != requests.codes.OK:
print("Request Error, Code: %d"% resp.status_code)
sys.exit()
soup = BeautifulSoup(resp.text,"html.parser")
contents = soup.find("div", id="contents")
wallpapers = contents.findAll("a", rel="wallpaper")
self.links = []
forwallpaperinwallpapers:
self.links.append(wallpaper['href'])
defdownloadImage(self):
ifos.path.exists("./Divnil") !=True:
os.mkdir("./Divnil")
forurlinself.links:
url = self.head + url
resp = requests.get(url, headers=self.headers)
ifresp.status_code != requests.codes.OK:
print("URL: %s REQUESTS ERROR. CODE: %d"% (url, resp.status_code))
continue
soup = BeautifulSoup(resp.text,"html.parser")
img = soup.find("div", id="contents").find("img", id="main_content")
img_url = self.head + img['original'].replace("../","")
img_name = img['alt']
print("start download %s ..."% img_url)
resp = requests.get(img_url, headers=self.headers)
ifresp.status_code != requests.codes.OK:
print("IMAGE %s DOWNLOAD FAILED."% img_name)
continue
if'/'inimg_name:
img_name = img_name.split('/')[1]
withopen("./Divnil/"+ img_name +".jpg","wb")asf:
f.write(resp.content)
defmain(self):
self.getImageInfoUrl()
self.downloadImage()
if__name__ =="__main__":
divnil = Divnil()
divnil.main()
学习中遇到问题有不明白的地方，推荐加小编python学习群：975188894内有视频教程 ，直播课程 ，等学习资料，期待你的加入！
-END-