使用activerecord-import 批量导入数据至mysql on rails enhanced active-record

Requiring

There are essentially two ways to require activerecord-import:

• Using Bundler (default Rails 3 app uses this)
• Not using Bundler

These instructions only work for activerecord-import 0.2.0 or higher. If you have a version before 0.2.0 then upgrade.

Using Bundler

There are two ways you can use Bundler. First, the Rails 3 way.

The Rails 3 Way – autoloading gem dependencies

Rails 3 by default autoloads your gem dependencies (thanks to config/application.rb). When this happens you only need to make sure activerecord-import is specified as a gem dependency in your Gemfile.

  gem "activerecord-import", ">= 0.2.0"

Not autoloading

If your gem dependencies aren’t autoloaded then simply require activerecord-import after activerecord has been loaded, ie:

require 'active_record'
require 'activerecord-import'

That’s it.

 

 

 

https://github.com/zdennis/activerecord-import

 

 

 

 

activerecord-import

activerecord-import is a library for bulk inserting data using ActiveRecord.

Why activerecord-import?

Because plain-vanilla, out-of-the-box ActiveRecord doesn’t provide support for inserting large amounts of data efficiently. With vanilla ActiveRecord you would have to perform individual save operations on each model:

10.times do |i|
  Book.create! :name => "book #{i}"
end

This may work fine if all you have is 10 records, but if you have hundreds, thousands, or millions of records it can turn into a nightmare. This is where activerecord-import comes into play.

An Introductory Example

To run examples on this page you’ll need to first require activerecord-import, for that please take a brief look at Requiring.

Here’s an example with equivalent behaviour using the #import method:

books = []
10.times do |i| 
  books << Book.new(:name => "book #{i}")
end
Book.import books

This call to import does whatever is most efficient for the underlying database adapter. Pretty slick, eh?

Features

Here’s a list of some of the high-level features that activerecord-import provides:

• activerecord-import can work with raw columns and arrays of values (fastest)
• activerecord-import works with model objects (faster)
• activerecord-import can perform validations (fast)
• activerecord-import can perform on duplicate key updates (requires mysql)

Upgrading from ar-extensions

activerecord-import replaces the ar-extensions library and is compatible with Rails 3. It provides the exact same API for importing data, but it does not include any additional ar-extensions functionality.]

 

 

 

 

 

 

desc "导入mail"

task :batch_import_email => :environment do

    require 'active_record'

    require 'activerecord-import'

    count=0

    count2=0

        r=[]

    File.open("/home/mlzboy/my/b2c2/doc/zzzzzzz.txt","r") do |file|

        file.each_line do |line|

            count2+=1

            begin

                if line.index("@").nil? ==false and line.index("zzzzzzz").nil?

                    email=line.chomp

                    #unless Subscription.exists?(:email=>email)

                        #Subscription.create(:email=>email)

                      r << Subscription.new(:email=>email)

                        count+=1

                        puts count

                    #end

                end

            rescue

                puts "==================="

                puts line

                puts count2

            end

            if r.size > 0 and r.size % 10000 == 0

                Subscription.import r

                r=[]

            end

        end

        if r.size!=0

            Subscription.import r

        end

    end

    put "finished!"

end