XMLFeedSpider例子

还是用xml的模板爬xml 来的直接。

TestItem in items.py:

class TestItem(scrapy.Item):
    id = scrapy.Field()
    name = scrapy.Field()
    description = scrapy.Field()

main:

from scrapy import log
from scrapy.contrib.spiders import XMLFeedSpider
from myproject.items import TestItem

class MySpider(XMLFeedSpider):
    name = 'example'
    start_urls = ['portfolio xii xml']
    iterator = 'xml' # This is actually unnecessary, since it's the default value
    itertag = 'PortfolioStatistics'  #Name of Parent Node

    def parse_node(self, response, node):
        item = TestItem()
        item['id'] = node.xpath('@_SalePosition').extract()  #Get Parent Node Attribute
        item['name'] = node.xpath('EquityStyle/EquityStyleCoreValue/StyleBox/text()').extract()  #Get son of grandson
        item['description'] = node.xpath('BondStatistics/EffectiveDuration/text()').extract() #Get Grandson
        return item

Output:
[
{“description”: [“0.53”], “name”: [“6”], “id”: [“L”]}
]

再进阶觉得这个例子不错，特别是pipeline那部分：
http://www.jianshu.com/p/e71ff6173302

2017-04-02：后续去抓带basic authentication的案例，会需要重写start_request才行。