Python爬虫：利用urlparse获取“干净”的url

urlparse 类似处理操作系统路径的 os.path 模块，能够很好的处理网址路径

导入模块

python3

from urllib.parse import urlparse, urljoin

python2

from urlparse import urlparse, urljoin

使用测试

url = "https://cdn.itjuzi.com/images/51202bf56a442ba934fe15d34a3f2976.png?imageView2/0/w/58/q/100"

ret = urlparse(url)
print ret
# ParseResult(scheme='https', netloc='cdn.itjuzi.com', 
# path='/images/51202bf56a442ba934fe15d34a3f2976.png',
# params='', query='imageView2/0/w/58/q/100', fragment='')

link = urljoin(ret.scheme+"://"+ret.netloc, ret.path)
print link
# https://cdn.itjuzi.com/images/51202bf56a442ba934fe15d34a3f2976.png

封装成函数

def get_clean_url(url):
    """
    获取干净的url链接
    :param
        url: {str} url链接
    :return: {str} 干净的url链接
    """
    ret = urlparse(url)
    link = urljoin(ret.scheme + "://" + ret.netloc, ret.path)
    return link

print(get_clean_url(url))
# https://cdn.itjuzi.com/images/51202bf56a442ba934fe15d34a3f2976.png