Mechanize

The Mechanize 2.0 Handbook

Mechanize is a library for automated interaction with web sites. For all intents and purposes, it acts like a web browser with no user interface. It downloads web pages, can click on links, fill out and submit forms, store cookies, etc. Mechanize is useful for automated crawling, testing and scraping of web sites.

However, Mechanize does have one down side: it doesn't support Javascript. Mechanize is a rather small piece of software, more or less wrapping an HTTP library and Nokogiri. Because of this, any Javascript on the pages downloaded will not execute, the DOM tree will not be updated and any links that need Javascript to appear or function will not work with Mechanize.