Frequently Asked Questions — Scrapy 0.15.1 documentation

Frequently Asked Questions — Scrapy 0.15.1 documentation

Frequently Asked Questions¶

How does Scrapy compare to BeautifulSoup or lxml?¶

BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is
an application framework for writing web spiders that crawl web sites and
extract data from them.

Scrapy provides a built-in mechanism for extracting data (called
selectors) but you can easily use BeautifulSoup
(or lxml) instead, if you feel more comfortable working with them. After
all, they’re just parsing libraries which can be imported and used from any
Python code.

In other words, comparing BeautifulSoup (or lxml) to Scrapy is like
comparing jinja2 to Django.

What Python versions does Scrapy support?¶

Scrapy runs in Python 2.6 and 2.7.

Does Scrapy work with Python 3.0?¶

No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment,
Scrapy works with Python 2.6 and 2.7.

See also

What Python versions does Scrapy support?.

Did Scrapy “steal” X from Django?¶

Probably, but we don’t like that word. We think Django is a great open source
project and an example to follow, so we’ve used it as an inspiration for
Scrapy.

We believe that, if something is already done well, there’s no need to reinvent
it. This concept, besides being one of the foundations for open source and free
software, not only applies to software but also to documentation, procedures,
policies, etc. So, instead of going through each problem ourselves, we choose
to copy ideas from those projects that have already solved them properly, and
focus on the real problems we need to solve.

We’d be proud if Scrapy serves as an inspiration for other projects. Feel free
to steal from us!

Does Scrapy work with HTTP proxies?¶

Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP
Proxy downloader middleware. See
HttpProxyMiddleware.

Scrapy crashes with: ImportError: No module named win32api¶

You need to install pywin32 because of this Twisted bug.

How can I simulate a user login in my spider?¶

See Using FormRequest.from_response() to simulate a user login.

Does Scrapy crawl in breath-first or depth-first order?¶

By default, Scrapy uses a LIFO queue for storing pending requests, which
basically means that it crawls in DFO order. This order is more convenient
in most cases. If you do want to crawl in true BFO order, you can do it by
setting the following settings:

DEPTH_PRIORITY = 1
SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue'
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'

My Scrapy crawler has memory leaks. What can I do?¶

See Debugging memory leaks.

Also, Python has a builtin memory leak issue which is described in
Leaks without leaks.

How can I make Scrapy consume less memory?¶

See previous question.

Can I use Basic HTTP Authentication in my spiders?¶

Yes, see HttpAuthMiddleware.

Why does Scrapy download pages in English instead of my native language?¶

Try changing the default Accept-Language request header by overriding the
DEFAULT_REQUEST_HEADERS setting.

Where can I find some example Scrapy projects?¶

See Examples.

Can I run a spider without creating a project?¶

Yes. You can use the runspider command. For example, if you have a
spider written in a my_spider.py file you can run it with:

scrapy runspider my_spider.py

See runspider command for more info.

I get “Filtered offsite request” messages. How can I fix them?¶

Those messages (logged with DEBUG level) don’t necessarily mean there is a
problem, so you may not need to fix them.

Those message are thrown by the Offsite Spider Middleware, which is a spider
middleware (enabled by default) whose purpose is to filter out requests to
domains outside the ones covered by the spider.

For more info see:
OffsiteMiddleware.

What is the recommended way to deploy a Scrapy crawler in production?¶

See Scrapy Service (scrapyd).

Can I use JSON for large exports?¶

It’ll depend on how large your output is. See this warning in JsonItemExporter
documentation.

Can I return (Twisted) deferreds from signal handlers?¶

Some signals support returning deferreds from their handlers, others don’t. See
the Built-in signals reference to know which ones.

What does the response status code 999 means?¶

999 is a custom reponse status code used by Yahoo sites to throttle requests.
Try slowing down the crawling speed by using a download delay of 2 (or
higher) in your spider:

class MySpider(CrawlSpider):

    name = 'myspider'

    DOWNLOAD_DELAY = 2

    # [ ... rest of the spider code ... ]

Or by setting a global download delay in your project with the
DOWNLOAD_DELAY setting.

Can I call pdb.set_trace() from my spiders to debug them?¶

Yes, but you can also use the Scrapy shell which allows you too quickly analyze
(and even modify) the response being processed by your spider, which is, quite
often, more useful than plain old pdb.set_trace().

For more info see Invoking the shell from spiders to inspect responses.

Simplest way to dump all my scraped items into a JSON/CSV/XML file?¶

To dump into a JSON file:

scrapy crawl myspider -o items.json -t json

To dump into a CSV file:

scrapy crawl myspider -o items.csv -t csv

To dump into a XML file:

scrapy crawl myspider -o items.xml -t xml

For more information see Feed exports

What’s this huge cryptic __VIEWSTATE parameter used in some forms?¶

The __VIEWSTATE parameter is used in sites built with ASP.NET/VB.NET. For
more info on how it works see this page. Also, here’s an example spider
which scrapes one of these sites.

What’s the best way to parse big XML/CSV data feeds?¶

Parsing big feeds with XPath selectors can be problematic since they need to
build the DOM of the entire feed in memory, and this can be quite slow and
consume a lot of memory.

In order to avoid parsing all the entire feed at once in memory, you can use
the functions xmliter and csviter from scrapy.utils.iterators
module. In fact, this is what the feed spiders (see Spiders) use
under the cover.

Does Scrapy manage cookies automatically?¶

Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them
back on subsequent requests, like any regular web browser does.

For more info see Requests and Responses and CookiesMiddleware.

How can I see the cookies being sent and received from Scrapy?¶

Enable the COOKIES_DEBUG setting.

How can I instruct a spider to stop itself?¶

Raise the CloseSpider exception from a callback. For
more info see: CloseSpider.

How can I prevent my Scrapy bot from getting banned?¶

Some websites implement certain measures to prevent bots from crawling them,
with varying degrees of sophistication. Getting around those measures can be
difficult and tricky, and may sometimes require special infrastructure.

Here are some tips to keep in mind when dealing with these kind of sites:

• rotate your user agent from a pool of well-known ones from browsers (google
  around to get a list of them)
• disable cookies (see COOKIES_ENABLED) as some sites may use
  cookies to spot bot behaviour
• use download delays (2 or higher). See DOWNLOAD_DELAY setting.
• is possible, use Google cache to fetch pages, instead of hitting the sites
  directly
• use a pool of rotating IPs. For example, the free Tor project.

If you are still unable to prevent your bot getting banned, consider contacting
commercial support.