给scrapy设置HTTP代理

在setting.py同级目录新建一个middlewares.py

class ProxyMiddleware(object):
    # overwrite process request
    def process_request(self, request, spider):
        # Set the location of the proxy
        request.meta['proxy'] = "http://YOUR_PROXY_IP:PORT"

然后再setting.py中添加

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'project_name.middlewares.ProxyMiddleware': 100,
}

即可

参考文章

http://mahmoud.abdel-fattah.net/2012/04/07/using-scrapy-with-proxies/