数据迁移备份的几种策略比较

   

Data migration Approaches:

Model1:

Advantages:

Simple

Disadvantages:

Business disrupted

Migrationneeds to complete in limited time

Model2:

Require each table to have an additional GMT_MODIFIED column

Advantages:

Shorter down time, data can be read from original DB when performing full data replication

Disadvantages

Hard to predict whether new DB can fully withstand all read/write load. If after switching to new DB, and it is overloaded, the result can be disastrous

Timestamp based scanning creates a lot of pressure on DB. If scan interval is too large, data is out of sync.

Model3:

Advantages:

Low data sync latency

No downtime

Disadvantages:

Intrusion into business logic to write to both database

Full replication and dual write occurs the same time, data in new DB may have conflict. We need to logically enforce that only records with newer timestamp are written to new DB

Date integrity may be comprosied: writing to old/new database need to be in same transaction. XA is very costly.

Incremental data replication is more widely adopted

For oracle it is harder to replicate journal, so use a two phase protocol

Write to a log table

Write to data table

Write data record id to log table

Asynchronously retrieve log based on id and replicate to new DB

Following modules are needed for data migration program

Control Module

Zookeeper

Timer

Task configuration

Full migration

Data fetch module

Customer defined functions

Data write module

Incremental migration

Data fetch module

Customer defined functions

Data write module

Incremental trigger

Data validation module

Model4:

Step 1:

Stop updated business data during full migration in local

Step 2:

Start full migration, need to consider exceptional cases where the target database   server is down unexpectedly during migration. Solution is to store id range to replicate in zookeeper, so that migration progress can be queried from zookeeper on target server recovery

Step 3:

Start incremental migration, replicate local stored update to target database server until all local updates are consumed

Step 4:

Start data validation to ensure that migrated data and original data are the same (no missing or duplicated data). We can do full validation or sample validation based on importance of data. After validation, we can tentatively switch some of the read to new DB

Step 5:

Swith to use the new DB. After switch, write will be applied to the new DB and it is very hard to rollback

Multi-threaded migration consideration

1. Mulit-threaded migration increases write speed, but consumes more DB resources. Number of reades need to be dynamically adjusted based on time (business load)

2. Multi-threaded migration may cause deadlock, need to ensure that per write thread  per table.

3. Data recovery becomes a challenge as it is more difficult to find out data failure point. The solution is to use a central dispatcher to allocate data range to migrate to different worker thread.