python爬网页上所有的链接（爬到最深）

相关课程链接：Crawl Web

今天做的这个是在上个实验的基础上加了一个跳转挖掘链接，再从新链接里面继续向下挖掘，这样层层递进挖到深处~~

还没有学到get_page的真正写法，如果用urllib2.urlopen()会出现HTTP error的问题，这个在第四章才学习。这里直接贴上网站的源码，主要验证深挖的函数。

def get_page(url):   #尚未处理好，功能是传入网址，返回源码
    try:
        if url == "http://xkcd.com/353":
            return  '           
 
 
 
 
 
 
\t
 
Archive
\t 
News/Blag
 
Store
 
About
 
Forums
 
 
 
 
 
 
 
 
 
 
 
 
 
  

A webcomic of romance,
 sarcasm, math, and language.
 
 
XKCD updates every Monday, Wednesday, and Friday. 
 
 
 
 
 
 
 
 
 
 
 
 
 
Python

 
 
|<
 
< Prev
 
Random
 
Next >
 
>|
 



 
 
|<
 
< Prev
 
Random
 
Next >
 
>|
 
Permanent link to this comic: http://xkcd.com/353/
Image URL (for hotlinking/embedding): http://imgs.xkcd.com/comics/python.png
[[ Guy 1 is talking to Guy 2, who is floating in the sky ]]Guy 1: You39;re flying! How?Guy 2: Python!Guy 2: I learned it last night! Everything is so simple!Guy 2: Hello world is just 39;print "Hello, World!" 39;Guy 1: I dunno... Dynamic typing? Whitespace?Guy 2: Come join us! Programming is fun again! It39;s a whole new world up here!Guy 1: But how are you flying?Guy 2: I just typed 39;import antigravity39;Guy 1: That39;s it?Guy 2: ...I also sampled everything in the medicine cabinet for comparison.Guy 2: But i think this is the python.{{ I wrote 20 short programs in Python yesterday. It was wonderful. Perl, I39;m leaving you. }}
 
 
 
 
 
 
 
 
 
 
  

Search comic titles and transcripts:
 
     
RSS Feed - Atom Feed
 
 
 Comics I enjoy:
 Dinosaur Comics, A Softer World, Perry Bible Fellowship, Copper, Questionable Content, Achewood, Wondermark, Indexed, Buttercup Festival 
 
 Warning: this comic occasionally contains strong language (which may be unsuitable for children), unusual humor (which may be unsuitable for adults), and advanced mathematics (which may be unsuitable for liberal-arts majors).
 
 
We did not invent the algorithm. The algorithm consistently finds Jesus. The algorithm killed Jeeves. 
The algorithm is banned in China. The algorithm is from Jersey. The algorithm constantly finds Jesus.
This is not the algorithm. This is close.
 
 
 
  This work is licensed under a Creative Commons Attribution-NonCommercial 2.5 License. 
 This means you\"re free to copy and share these comics (but not to sell them). More details.
 
 
 
 
 
 
 
 
  '
        elif url == "http://xkcd.com/554":
            return  '              
 
 
 
 
 
 
 
 
Archive
 
News/Blag
 
Store
 
About
 
Forums
 
 
 
 
 
 
 
 
 
 
 
 
 
  

A webcomic of romance,
 sarcasm, math, and language.
 
 XKCD updates every Monday, Wednesday, and Friday. 
 Blag: Remember geohashing? Something pretty cool happened Sunday. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Not Enough Work
 
 
 
 
|<
 
< Prev
 
Random
 
Next >
 
>|
 
 
 
 
 
 
 
 
 
|<
 
< Prev
 
Random
 
Next >
 
>|
 
 
 
Permanent link to this comic: http://xkcd.com/554/
 
Image URL (for hotlinking/embedding): http://imgs.xkcd.com/comics/not_enough_work.png
 
Narration: Signs your coders don39;t have enough work to do: [[A man sitting at his workstation; a female co-worker behind him]] Man: I39;m almost up to my old typing speed in dvorak [[Two men standing by a server rack]] Man  1: Our servers now support gopher. Man  1: Just in case. [[A woman standing near her workstation speaking to a male co-worker]] Woman: Our pages are now HTML, XHTML-STRICT, and haiku-compliant Man: Haiku? Woman: 
 Woman:  Woman: Blog!< span>< div> [[A woman sitting at her workstation]] Woman: Hey! Have you guys seen this webcomic? {{title text: It39;s even harder if you39;re an asshole who pronounces <> brackets.}}
 
 
 
 
 
 
 
 
 
 
  

 Search comic titles and transcripts:
   
 
     
 
  RSS Feed - Atom Feed 
 
 
 Comics I enjoy:
 Three Word Phrase, Oglaf (nsfw), SMBC, Dinosaur Comics, A Softer World, Buttersafe, Perry Bible Fellowship, Questionable Content, Buttercup Festival 
 
 Warning: this comic occasionally contains strong language (which may be unsuitable for children), unusual humor (which may be unsuitable for adults), and advanced mathematics (which may be unsuitable for liberal-arts majors).
 
 
We did not invent the algorithm. The algorithm consistently finds Jesus. The algorithm killed Jeeves. 
The algorithm is banned in China. The algorithm is from Jersey. The algorithm constantly finds Jesus.
This is not the algorithm. This is close.
 
 
 
  This work is licensed under a Creative Commons Attribution-NonCommercial 2.5 License.  
 This means you"re free to copy and share these comics (but not to sell them). More details.
 
 
 
 
 
 
 
 
   '
    except:
        return ""
    return ""

def get_next_target(page):   #对源码进行查询链接功能，返回链接和该链接结束位置
    start_link = page.find('

总之主体大意就是，对当前网页进行挖掘，返回所有子链接，添加到未挖掘队列

每一次对未挖掘队列中的一个元素进行挖掘，继续将查找到的子链接添加到未挖掘队列，直到未挖掘队列为空

每次查询就将已挖掘的记录下来，最后返回已挖掘队列，并在操作过程中有个查重的功能。