2013
-
12
-
12
16
:
57
:
06
+
0530
[scrapy] INFO: Scrapy
0.20
.
2
started (bot: hn)
2013
-
12
-
12
16
:
57
:
06
+
0530
[scrapy] DEBUG: Optional features available: ssl, http11, django
2013
-
12
-
12
16
:
57
:
06
+
0530
[scrapy] DEBUG: Overridden settings: {
'NEWSPIDER_MODULE'
:
'hn.spiders'
,
'SPIDER_MODULES'
: [
'hn.spiders'
],
'BOT_NAME'
:
'hn'
}
2013
-
12
-
12
16
:
57
:
06
+
0530
[scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2013
-
12
-
12
16
:
57
:
06
+
0530
[scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware
, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013
-
12
-
12
16
:
57
:
06
+
0530
[scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013
-
12
-
12
16
:
57
:
06
+
0530
[scrapy] DEBUG: Enabled item pipelines:
2013
-
12
-
12
16
:
57
:
06
+
0530
[hn] INFO: Spider opened
2013
-
12
-
12
16
:
57
:
06
+
0530
[hn] INFO: Crawled
0
pages (at
0
pages
/
min
), scraped
0
items (at
0
items
/
min
)
2013
-
12
-
12
16
:
57
:
06
+
0530
[scrapy] DEBUG: Telnet console listening on
0.0
.
0.0
:
6023
2013
-
12
-
12
16
:
57
:
06
+
0530
[scrapy] DEBUG: Web service listening on
0.0
.
0.0
:
6080
2013
-
12
-
12
16
:
57
:
07
+
0530
[hn] DEBUG: Redirecting (
301
) to
/
/
news.ycombinator.com
/
>
from
/
/
news.ycombinator.com>
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Crawled (
200
)
/
/
news.ycombinator.com
/
> (referer:
None
)
(u
'Caltech Announces Open Access Policy | Caltech'
, u
'http://www.caltech.edu/content/caltech-announces-open-access-policy'
)
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Scraped
from
<
200
https:
/
/
news.ycombinator.com
/
>
{
'link'
: u
'http://www.caltech.edu/content/caltech-announces-open-access-policy'
,
'title'
: u
'Caltech Announces Open Access Policy | Caltech'
}
(u
'Coinbase Raises $25 Million From Andreessen Horowitz'
, u
'http://blog.coinbase.com/post/69775463031/coinbase-raises-25-million-from-andreessen-horowitz'
)
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Scraped
from
<
200
https:
/
/
news.ycombinator.com
/
>
{
'link'
: u
'http://blog.coinbase.com/post/69775463031/coinbase-raises-25-million-from-andreessen-horowitz'
,
'title'
: u
'Coinbase Raises $25 Million From Andreessen Horowitz'
}
(u
'Backpacker stripped of tech gear at Auckland Airport'
, u
'http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11171475'
)
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Scraped
from
<
200
https:
/
/
news.ycombinator.com
/
>
{
'link'
: u
'http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11171475'
,
'title'
: u
'Backpacker stripped of tech gear at Auckland Airport'
}
(u
'How I introduced a 27-year-old computer to the web'
, u
'http://www.keacher.com/1216/how-i-introduced-a-27-year-old-computer-to-the-web/'
)
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Scraped
from
<
200
https:
/
/
news.ycombinator.com
/
>
{
'link'
: u
'http://www.keacher.com/1216/how-i-introduced-a-27-year-old-computer-to-the-web/'
,
'title'
: u
'How I introduced a 27-year-old computer to the web'
}
(u
'Show HN: Bitcoin Pulse - Tracking Bitcoin Adoption'
, u
'http://www.bitcoinpulse.com'
)
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Scraped
from
<
200
https:
/
/
news.ycombinator.com
/
>
{
'link'
: u
'http://www.bitcoinpulse.com'
,
'title'
: u
'Show HN: Bitcoin Pulse - Tracking Bitcoin Adoption'
}
(u
'Why was this secret?'
, u
'http://sivers.org/ws'
)
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Scraped
from
<
200
https:
/
/
news.ycombinator.com
/
>
{
'link'
: u
'http://sivers.org/ws'
,
'title'
: u
'Why was this secret?'
}
(u
'PostgreSQL Exercises'
, u
'http://pgexercises.com/'
)
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Scraped
from
<
200
https:
/
/
news.ycombinator.com
/
>
{
'link'
: u
'http://pgexercises.com/'
,
'title'
: u
'PostgreSQL Exercises'
}
(u
'What it feels like being an ipad on a stick on wheels'
, u
'http://labs.spotify.com/2013/12/12/what-it-feels-like-being-an-ipad-on-a-stick-on-wheels/'
)
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Scraped
from
<
200
https:
/
/
news.ycombinator.com
/
>
{
'link'
: u
'http://labs.spotify.com/2013/12/12/what-it-feels-like-being-an-ipad-on-a-stick-on-wheels/'
,
'title'
: u
'What it feels like being an ipad on a stick on wheels'
}
(u
'Prototype ergonomic mechanical keyboards'
, u
'http://blog.fsck.com/2013/12/better-and-better-keyboards.html'
)
2013
-
12
-
12
16
:
57
:
08
+
0530
[hn] DEBUG: Scraped
from
<
200
https:
/
/
news.ycombinator.com
/
>
{
'link'
: u
'http://blog.fsck.com/2013/12/better-and-better-keyboards.html'
,
'title'
: u
'Prototype ergonomic mechanical keyboards'
}
(u
'H5N1'
, u
'http://blog.samaltman.com/h5n1'
)
.............
.............
.............
2013
-
12
-
12
16
:
58
:
41
+
0530
[hn] INFO: Closing spider (finished)
2013
-
12
-
12
16
:
58
:
41
+
0530
[hn] INFO: Dumping Scrapy stats:
{
'downloader/exception_count'
:
2
,
'downloader/exception_type_count/twisted.internet.error.DNSLookupError'
:
2
,
'downloader/request_bytes'
:
22401
,
'downloader/request_count'
:
71
,
'downloader/request_method_count/GET'
:
71
,
'downloader/response_bytes'
:
1482842
,
'downloader/response_count'
:
69
,
'downloader/response_status_count/200'
:
61
,
'downloader/response_status_count/301'
:
4
,
'downloader/response_status_count/302'
:
3
,
'downloader/response_status_count/404'
:
1
,
'finish_reason'
:
'finished'
,
'finish_time'
: datetime.datetime(
2013
,
12
,
12
,
11
,
28
,
41
,
289000
),
'item_scraped_count'
:
63
,
'log_count/DEBUG'
:
141
,
'log_count/INFO'
:
4
,
'request_depth_max'
:
2
,
'response_received_count'
:
62
,
'scheduler/dequeued'
:
71
,
'scheduler/dequeued/memory'
:
71
,
'scheduler/enqueued'
:
71
,
'scheduler/enqueued/memory'
:
71
,
'start_time'
: datetime.datetime(
2013
,
12
,
12
,
11
,
27
,
6
,
843000
)}
2013
-
12
-
12
16
:
58
:
41
+
0530
[hn] INFO: Spider closed (finished)
