Elixir爬虫 — crawlie简介

https://github.com/nietaki/crawlie

Crawlie 使用了 Elixir 的 GenStage 功能实现了并行爬虫。大部分工作通过 Crawlie.Stage.UrlManager 来完成。consume 用户提供的 url 列表，receive 接下来处理得到的 url，并保证每个 url 只被处理一次，保证 discovered urls 集合尽可能小，通过深度优先查找 url 树。

urls 通过 Crawlie.Stage.UrlManager 获取，GenStage Flow，然后使用 HTTPoison 并行获取 urls，并使用用户提供的回调函数处理响应。被发现的 urls 被发回 UrlManager。

下面是它的流程图

如果你对爬虫的统计数据感兴趣，可以查看Crawlie.crawl_and_track_stats/3
。它开启了一个Stats GenServer
在 Crawlie's supervision tree。收集爬虫过程的数据。

这个扩展通过如下方式使用：

Add crawlie to your list of dependencies in mix.exs:

def deps do
  [{:crawlie, "~> 0.6.0"}]
end

Ensure crawlie is started before your application:

def application do
  [applications: [:crawlie]]
end

看看示例

https://github.com/nietaki/crawlie_example

$ mix deps.get
$ mix crawlie.example

most popular words longer than 5 letters in the vicinity of ["https://en.wikipedia.org/wiki/Elixir_(programming_language)", "https://en.wikipedia.org/wiki/Mainframe_computer"]:
{word, count}
=============
{"system", 1973}
{"computer", 1618}
{"systems", 1257}
{"programming", 1165}
{"language", 1147}
{"software", 1052}
{"operating", 1022}
{"computers", 887}
{"languages", 873}
{"program", 825}
{"memory", 814}
{"number", 798}
{"called", 767}
{"between", 724}
{"company", 693}
{"support", 678}
{"different", 649}
{"including", 623}
{"however,", 620}
{"control", 590}

运行结果

源代码主要看这两个文件：
lib/mix/tasks/crawlie/example.ex and lib/crawlie_example/word_count_logic.ex.

https://github.com/nietaki/crawlie