爬取58同城，解决反爬字体加密解析问题

【导语】我们在爬取数据中，会遇到字体乱码的下，其实是字体加密，本篇文章主要解决字体解密这种反爬方式。

1.在浏览器打开58同城网址进入北京租房
2.点击检查,找到房租价格对应位置,发现源码中价格部分是乱码,但是页面显示正常
3.得出结果这段代码是经过字体加密的

我们看到的如下：

image.png

那么，按F12查看源码，如下：

image.png

【解决思路】

字体加密一般是通过字体文件进行映射,所以我们只需要找到对应的字体文件,然后通过一些手段找到他们字体对应的映射关系就可以破解字体反爬

1、当我们发现字体反爬之后,首先可以找一下是通过哪一个类进行的字体映射,因为字体映射都会在具体的css样式上进行体现
2、我们可以先删除掉自身的一些样式,观察页面变化,当我们删除自身样式或者父类样式之后页面恢复了乱码,就可以确定是哪一个类进行的字体加密

image.png

3、通过加密样式寻找字体映射文件方式

字体加密一般是分为两种
①通过base64加密的字体文件,这种文件可以通过python的base64模块进行解密
②通过url的方式引入字体文件

以上两种我们都可以通过搜索对应的加密样式找到具体文件
首先可以在当前页面中搜索是否存在对这个类的定义
如果没有找到可以在network中全局搜索存在这个样式的css文件
找到加密文件之后无论是哪一种加密方式都可以通过python请求获取加密字体文件

image.png

4、获取加密文件的代码

通过正则表达式获取加密文件内容
获取到加密文件之后如果想要看里面的内容可以通过base64进行解密,并写入woff文件中
之后可以通过fontTools包将字体文件转为xml进行查看

import re
from fontTools.ttLib import TTFont
import base64

import requests

res = requests.get('https://bj.58.com/chuzu/?utm_source=market&spm=u-2d2yxv86y3v43nkddh1.BDPCPZ_BT&PGTID=0d100000-0000-13e1-612a-4575bfeb5389&ClickID=2')

bs64 = re.findall(';base64,(.*?)\'\) format',res.text,re.S)[0]
print(bs64)

image.png

import re
from fontTools.ttLib import TTFont
import base64

import requests

res = requests.get('https://bj.58.com/chuzu/?utm_source=market&spm=u-2d2yxv86y3v43nkddh1.BDPCPZ_BT&PGTID=0d100000-0000-13e1-612a-4575bfeb5389&ClickID=2')

bs64 = re.findall(';base64,(.*?)\'\) format',res.text,re.S)[0]
def base64tofile(bs64):
    res = base64.b64decode(bs64)
    f = open('1.woff', 'wb')
    f.write(res)
    f.close()
    font = TTFont('1.woff')
    font.saveXML('1.xml')
base64tofile(bs64)

打开xml进行查看时最重要的内容时cmap 这里是映射关系
code是源码中的内容
code 是16进制数,转成int类型之后代表Unicode上的一个对应的字符
name 是对应的页面中显示的内容
name是通过坐标进行绘制的图形,坐标信息在TTGlyph中展示
但是如果单单是数字加密,都可以找出期中的规律,如果是文字也进行了加密,那么只能通过一些工具查看绘制出的字体样式,手动的进行对应

image.png

5、 解密代码的编辑

1. 通过找到的对应关系进行解密代码的编辑
   通过minidom读取xml输入提前cmap中的对应关系
2. 通过re模块获取所有的价格数据,通过编写好的解密代码进行解密

from xml.dom import minidom
name_to_num = {'glyph00008':'7',
                'glyph00006':'5',
                'glyph00005':'4',
                'glyph00004':'3',
                'glyph00010':'9',
                'glyph00003':'2',
                'glyph00009':'8',
                'glyph00001':'0',
                'glyph00002':'1',
                'glyph00007':'6'}

def get_num(string):
    dom = minidom.parse('1.xml')
    root = dom.documentElement
    map_ = root.getElementsByTagName('cmap')[0].getElementsByTagName('map')
    code_to_name = {tag.getAttribute("code")[2:]:tag.getAttribute("name") for tag in map_[:]}
    print(code_to_name)
    nums = []
    for i in string.split(';'):
        if not i:
            break
        name_num = code_to_name[i[3:]]
        num = name_to_num[name_num]
        nums.append(num)
    return ''.join(nums)

# get_num(code_64)
get_num('龤驋龥龒')

image.png

【测试】

a = 'AAEAAAALAIAAAwAwR1NVQiCLJXoAAAE4AAAAVE9TLzL4XQjtAAABjAAAAFZjbWFwq8V/YgAAAhAAAAIuZ2x5ZuWIN0cAAARYAAADdGhlYWQbKimFAAAA4AAAADZoaGVhCtADIwAAALwAAAAkaG10eC7qAAAAAAHkAAAALGxvY2ED7gSyAAAEQAAAABhtYXhwARgANgAAARgAAAAgbmFtZTd6VP8AAAfMAAACanBvc3QEQwahAAAKOAAAAEUAAQAABmb+ZgAABLEAAAAABGgAAQAAAAAAAAAAAAAAAAAAAAsAAQAAAAEAAN6JRZhfDzz1AAsIAAAAAADb0W86AAAAANvRbzoAAP/mBGgGLgAAAAgAAgAAAAAAAAABAAAACwAqAAMAAAAAAAIAAAAKAAoAAAD/AAAAAAAAAAEAAAAKADAAPgACREZMVAAObGF0bgAaAAQAAAAAAAAAAQAAAAQAAAAAAAAAAQAAAAFsaWdhAAgAAAABAAAAAQAEAAQAAAABAAgAAQAGAAAAAQAAAAEERAGQAAUAAAUTBZkAAAEeBRMFmQAAA9cAZAIQAAACAAUDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFBmRWQAQJR2n6UGZv5mALgGZgGaAAAAAQAAAAAAAAAAAAAEsQAABLEAAASxAAAEsQAABLEAAASxAAAEsQAABLEAAASxAAAEsQAAAAAABQAAAAMAAAAsAAAABAAAAaYAAQAAAAAAoAADAAEAAAAsAAMACgAAAaYABAB0AAAAFAAQAAMABJR2lY+ZPJpLnjqeo59kn5Kfpf//AACUdpWPmTyaS546nqOfZJ+Sn6T//wAAAAAAAAAAAAAAAAAAAAAAAAABABQAFAAUABQAFAAUABQAFAAUAAAABgAIAAcABAAKAAMACQACAAEABQAAAQYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAiAAAAAAAAAAKAACUdgAAlHYAAAAGAACVjwAAlY8AAAAIAACZPAAAmTwAAAAHAACaSwAAmksAAAAEAACeOgAAnjoAAAAKAACeowAAnqMAAAADAACfZAAAn2QAAAAJAACfkgAAn5IAAAACAACfpAAAn6QAAAABAACfpQAAn6UAAAAFAAAAAAAAACgAPgBmAJoAvgDoASQBOAF+AboAAgAA/+YEWQYnAAoAEgAAExAAISAREAAjIgATECEgERAhIFsBEAECAez+6/rs/v3IATkBNP7S/sEC6AGaAaX85v54/mEBigGB/ZcCcwKJAAABAAAAAAQ1Bi4ACQAAKQE1IREFNSURIQQ1/IgBW/6cAicBWqkEmGe0oPp7AAEAAAAABCYGJwAXAAApATUBPgE1NCYjIgc1NjMyFhUUAgcBFSEEGPxSAcK6fpSMz7y389Hym9j+nwLGqgHButl0hI2wx43iv5D+69b+pwQAAQAA/+YEGQYnACEAABMWMzI2NRAhIzUzIBE0ISIHNTYzMhYVEAUVHgEVFAAjIiePn8igu/5bgXsBdf7jo5CYy8bw/sqow/7T+tyHAQN7nYQBJqIBFP9uuVjPpf7QVwQSyZbR/wBSAAACAAAAAARoBg0ACgASAAABIxEjESE1ATMRMyERNDcjBgcBBGjGvv0uAq3jxv58BAQOLf4zAZL+bgGSfwP8/CACiUVaJlH9TwABAAD/5gQhBg0AGAAANxYzMjYQJiMiBxEhFSERNjMyBBUUACEiJ7GcqaDEx71bmgL6/bxXLPUBEv7a/v3Zbu5mswEppA4DE63+SgX42uH+6kAAAAACAAD/5gRbBicAFgAiAAABJiMiAgMzNjMyEhUUACMiABEQACEyFwEUFjMyNjU0JiMiBgP6eYTJ9AIFbvHJ8P7r1+z+8wFhASClXv1Qo4eAoJeLhKQFRj7+ov7R1f762eP+3AFxAVMBmgHjLfwBmdq8lKCytAAAAAABAAAAAARNBg0ABgAACQEjASE1IQRN/aLLAkD8+gPvBcn6NwVgrQAAAwAA/+YESgYnABUAHwApAAABJDU0JDMyFhUQBRUEERQEIyIkNRAlATQmIyIGFRQXNgEEFRQWMzI2NTQBtv7rAQTKufD+3wFT/un6zf7+AUwBnIJvaJLz+P78/uGoh4OkAy+B9avXyqD+/osEev7aweXitAEohwF7aHh9YcJlZ/7qdNhwkI9r4QAAAAACAAD/5gRGBicAFwAjAAA3FjMyEhEGJwYjIgA1NAAzMgAREAAhIicTFBYzMjY1NCYjIga5gJTQ5QICZvHD/wABGN/nAQT+sP7Xo3FxoI16pqWHfaTSSgFIAS4CAsIBDNbkASX+lf6l/lP+MjUEHJy3p3en274AAAAAABAAxgABAAAAAAABAA8AAAABAAAAAAACAAcADwABAAAAAAADAA8AFgABAAAAAAAEAA8AJQABAAAAAAAFAAsANAABAAAAAAAGAA8APwABAAAAAAAKACsATgABAAAAAAALABMAeQADAAEECQABAB4AjAADAAEECQACAA4AqgADAAEECQADAB4AuAADAAEECQAEAB4A1gADAAEECQAFABYA9AADAAEECQAGAB4BCgADAAEECQAKAFYBKAADAAEECQALACYBfmZhbmdjaGFuLXNlY3JldFJlZ3VsYXJmYW5nY2hhbi1zZWNyZXRmYW5nY2hhbi1zZWNyZXRWZXJzaW9uIDEuMGZhbmdjaGFuLXNlY3JldEdlbmVyYXRlZCBieSBzdmcydHRmIGZyb20gRm9udGVsbG8gcHJvamVjdC5odHRwOi8vZm9udGVsbG8uY29tAGYAYQBuAGcAYwBoAGEAbgAtAHMAZQBjAHIAZQB0AFIAZQBnAHUAbABhAHIAZgBhAG4AZwBjAGgAYQBuAC0AcwBlAGMAcgBlAHQAZgBhAG4AZwBjAGgAYQBuAC0AcwBlAGMAcgBlAHQAVgBlAHIAcwBpAG8AbgAgADEALgAwAGYAYQBuAGcAYwBoAGEAbgAtAHMAZQBjAHIAZQB0AEcAZQBuAGUAcgBhAHQAZQBkACAAYgB5ACAAcwB2AGcAMgB0AHQAZgAgAGYAcgBvAG0AIABGAG8AbgB0AGUAbABsAG8AIABwAHIAbwBqAGUAYwB0AC4AaAB0AHQAcAA6AC8ALwBmAG8AbgB0AGUAbABsAG8ALgBjAG8AbQAAAAIAAAAAAAD/EwB3AAAAAAAAAAAAAAAAAAAAAAAAAAAACwECAQMBBAEFAQYBBwEIAQkBCgELAQwAAAAAAAAAAAAAAAAAAAAA'
# '龒鸺鸺龤' = 1990
s = '龒鸺鸺龤'
base64tofile(a)

def str_to_16(s):
    code_64 = []
    for i in s:
        # 字体转unicode 对应编码位置
        decode_num = ord(i)
        # 转成16进制
        num =  hex(decode_num)
        code_64.append('$%s'%num)
    return ';'.join(code_64)

code_64 = str_to_16('龒鸺鸺龤')

get_num(code_64)

image.png

希望本文的内容对大家的学习或者工作能带来一定的帮助，每天进步一点点，加油