R语言rvest包使用

R中有好几个包都可以抓取网页数据，但是rvest + CSS Selector最方便。

通过查看器立刻知道表格数据都在td:nth-child(1),td:nth-child(3)之类的节点中，直接代码提取就行了。

library(rvest)

先看看都有什么

freak <- html_session("http://torrentfreak.com/top-10-most-pirated-movies-of-the-week-130304/")

freak

<session> http://torrentfreak.com/top-10-most-pirated-movies-of-the-week-130304/

 Status: 200

 Type:   text/html; charset=UTF-8

 Size:   24983

 

freak %>% html_nodes("td:nth-child(3)") %>% html_text() %>% .[1:10]

[1] "Silver Linings Playbook "          

[2] "The Hobbit: An Unexpected Journey "

[3] "Life of Pi (DVDscr/DVDrip)"        

[4] "Argo (DVDscr)"                    

[5] "Identity Thief "                  

[6] "Red Dawn "                        

[7] "Rise Of The Guardians (DVDscr)"    

[8] "Django Unchained (DVDscr)"        

[9] "Lincoln (DVDscr)"                  

[10] "Zero Dark Thirty "  

 

freak %>% html_nodes("td:nth-child(1)") %>% html_text() %>% .[2:11]

[1] "1"  "2"  "3"  "4"  "5"  "6"  "7"  "8"  "9"  "10"

 

freak %>% html_nodes("td:nth-child(4)") %>% html_text() %>% .[1:10]

[1] "7.4 / trailer" "8.2 / trailer" "8.3 / trailer" "8.2 / trailer"

[5] "8.2 / trailer" "5.3 / trailer" "7.5 / trailer" "8.8 / trailer"

[9] "8.2 / trailer" "7.6 / trailer"

 

freak %>% html_nodes("td:nth-child(4) a[href*='imdb']") %>% html_attr("href") %>% .[1:10]

[1] "http://www.imdb.com/title/tt1045658/"

[2] "http://www.imdb.com/title/tt0903624/"

[3] "http://www.imdb.com/title/tt0454876/"

[4] "http://www.imdb.com/title/tt1024648/"

[5] "http://www.imdb.com/title/tt2024432/"

[6] "http://www.imdb.com/title/tt1234719/"

[7] "http://www.imdb.com/title/tt1446192/"

[8] "http://www.imdb.com/title/tt1853728/"

[9] "http://www.imdb.com/title/tt0443272/"

[10] "http://www.imdb.com/title/tt1790885/?"

#构建数据框

data.frame(movie=freak %>% html_nodes("td:nth-child(3)") %>% html_text() %>% .[1:10],

           rank=freak %>% html_nodes("td:nth-child(1)") %>% html_text() %>% .[2:11],

           rating=freak %>% html_nodes("td:nth-child(4)") %>% html_text() %>% .[1:10],

           imdb.url=freak %>% html_nodes("td:nth child(4) a[href*='imdb']") %>% html_attr("href") %>% .[1:10],stringsAsFactors=FALSE)

                               movie rank        rating                              imdb.url

1            Silver Linings Playbook     1 7.4 / trailer  http://www.imdb.com/title/tt1045658/

2  The Hobbit: An Unexpected Journey     2 8.2 / trailer  http://www.imdb.com/title/tt0903624/

3          Life of Pi (DVDscr/DVDrip)    3 8.3 / trailer  http://www.imdb.com/title/tt0454876/

4                       Argo (DVDscr)    4 8.2 / trailer  http://www.imdb.com/title/tt1024648/

5                     Identity Thief     5 8.2 / trailer  http://www.imdb.com/title/tt2024432/

6                           Red Dawn     6 5.3 / trailer  http://www.imdb.com/title/tt1234719/

7      Rise Of The Guardians (DVDscr)    7 7.5 / trailer  http://www.imdb.com/title/tt1446192/

8           Django Unchained (DVDscr)    8 8.8 / trailer  http://www.imdb.com/title/tt1853728/

9                    Lincoln (DVDscr)    9 8.2 / trailer  http://www.imdb.com/title/tt0443272/

10                  Zero Dark Thirty    10 7.6 / trailer  http://www.imdb.com/title/tt1790885/?

 

如果不考虑网址，还有更简单的方式：

 

freak %>% html_nodes("table") %>% html_table()

[[1]]

           Ranking (last week)                             Movie IMDb Rating / Trailer

1  torrentfreak.com        <NA>                              <NA>                  <NA>

2                 1         (5)           Silver Linings Playbook         7.4 / trailer

3                 2      (back) The Hobbit: An Unexpected Journey         8.2 / trailer

4                 3         (9)        Life of Pi (DVDscr/DVDrip)         8.3 / trailer

5                 4      (back)                     Argo (DVDscr)         8.2 / trailer

6                 5         (…)                    Identity Thief         8.2 / trailer

7                 6         (1)                          Red Dawn         5.3 / trailer

8                 7         (2)    Rise Of The Guardians (DVDscr)         7.5 / trailer

9                 8         (4)         Django Unchained (DVDscr)         8.8 / trailer

10                9         (6)                  Lincoln (DVDscr)         8.2 / trailer

11               10      (back)                  Zero Dark Thirty         7.6 / trailer