17. WEB APPLICATION

构建抽象层,然后逐层往下细化。自顶向下。分优先级。

爬虫应用:

接受一个网页,PARSE 内容,根据里面的HTML生成新的任务。

如何去重任务? 单机的话,数据量小可以用SET,大的话LOCAL set DB去重。多台机器用URL的HASH去做PARTITION。

写BFS code for a simple web crawler
multi thread version
to design a more practical system

瓶颈:
网络带宽
计算资源去PARSE网页。

17. WEB APPLICATION_第1张图片
image.png

队列SHARDED BY URL。 每个爬虫只去一个队列取任务。

常规PATTERN

1.经典方法去分发JOB给不同WORKER
shard with (hash, range)
batch - group thing together

2.hash table

3.push/pull model

4.handle failure
master/salve
replication

5.cache
write through(写CACHE,DB在一起,都完成再返回写完成)
write around (先写DB,当读CACHE MISS的时候,LOAD INTO CACHE)
write back

  1. database
    ACID
    sharding

7.knwoledge
network -> disk -> in memory -> l2 cache -> l1 cache
gist.github.com/jboner/2841832/

  1. synchronous/ Asynchronous call
    blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/

  2. stateful vs stateless
    stateless service is great because can be replicated as many as you want - best scenario for scaliablity - perfect for computational task
    easy to handle client retry

    sharded stateful service is hard to bring up new instance. because you need to redistribute data/ internal states

highscalability.com
aosabook.org/en/distsys.html

你可能感兴趣的:(17. WEB APPLICATION)