IBM copy service--ERM

之前讲述了IBM 三种copy service中的两种，Flashcopy，Volumecopy

IBM Copy Service--Flashcopy Introduction-http://taotao1240.blog.51cto.com/731446/1002206

IBM copy service--flashcopy 实验--http://taotao1240.blog.51cto.com/731446/1007957

IBM Copy Service--VolumeCopy--http://taotao1240.blog.51cto.com/731446/1009501

本篇讲述第三种copy service，ERM，即Enhanced Remote Mirror

同样地，先来看下redbook中关于ERM的定义：

ERM（Enhanced Remote Mirror） is the premium feature that enables the midrange storage subsystem to create a full image mirror of its logical drives on a second midrange storage subsystem, andkeep both images synchronized with one another.

ERM is used by customers to migrate their data from one midrange storage subsystem to another, or as the Remote Mirroring engine for their disaster recovery (DR) solution between two site locations across varying distances.

根据距离和网络的不同，ERM支持3种镜像方式：

--Metro mirror method(同步，数据要都写到local和remote，才算写成功)

Metro mirroring is a synchronous mirroring method, and is used for a campus or local distances (up to 10 miles). Any host write request is written to the primary (local) storage subsystem and transferred to the secondary (remote) storage subsystem. The host application must wait until receiving acknowledgement that the write request has been executed on both (local and remote) storage controllers.

当primary controller接收到主机的写请求，controller会首先写日志到mirror repository logical drive。同时，写数据到primary lun。接着Controller初始化写操作以复制改变的数据到secondary lun。当remote 的写操作完成时，primary controller从mirror repository lun上移除log。最终，controller给主机发消息说IO完成！

1、repository lun 只存储status和控制信息，而不存储真实的主机数据！

2、当写缓存被设置时，只要数据写到缓存中，IO操作就算完成；当两侧有一个写缓存被使用时，IO操作都必须等那一侧的IO操作完成之后才发送complete。

3、当controller接收到主机过来的读请求时，读请求被primary handle。primary和secondary之间没有通信！

--Global mirror method(异步)

Global mirroring is an asynchronous method of mirroring that uses write order consistency (WOC) to ensure that the I/O changes are sent to the secondary subsystem in the same sequence they were processed on the primary. This method must be used when the host application has its critical data, which is to be mirrored, spread across several logical drives. This method writes all requests from a host to the primary (local) storage subsystem and immediately reports them as completed to the host. The WOC ensures that the inter-dependent write requests are carried out in the same order at the remote site as they were performed at primary (local) site. It performs the mirroring writes in the same order that they were received. This is performed through the use of aconsistency group that tracks the order the member logical drive’s I/Os were received and processed.

--Global copy method（异步）

Global copy is a second asynchronous ERM write method. With this method, all write requests from a host are written to its primary (local) storage subsystem and are immediately reported to the host system as completed, as is the case with the Global Mirroring method. The secondary (target) is also written to at a later time. No inter-dependency checking is done to the other logical drives to determine whether relationships exist there. This method of mirroring can be used to copy static logical drives that have no I/O being performed to them（适合没有IO行为的静态逻辑驱动器）. It also can be used to mirror logical unit numbers (LUNs) that are totally independent from all other logical drives that are being mirrored

同步的方式比较好理解，但异步又分为两种，Global mirror method 和 Global copy method，根据redbook的定义，Global mirror 比Global copy多了个叫做consistency group的东东，字面上叫做“一致性组”，它跟踪IO被primary storage 接收和处理的顺序，然后根据这个顺序再在secondary storage 上执行。而Global copy则没有这个概念，也就无法按顺序执行，因此适用于那些没有IO行为的LUN。

Tips：

a、如果primary 是FC盘，而secondary 是SATA盘，则最好secondary 的SATA盘数量多点，以达到好的性能。一个成功的DR solution 是不能有很慢的secondary！

b、将ERM repository 放到快速磁盘上（15000rpm），不要有大的IO负载。这能降低ERM 操作的影响，尤其在同步操作时和使用consistency group时。

c、如果在高事务环境中mirror 重要数据，secondary 最好使用和primary相同的驱动器类型

d、假如两地距离100Km，而光的传输速度是200000km/s，因此从A地传输至B地需要0.0005s。一个IO是round-trip，因此一个IO需要0.0005s*2=0.001s。则最大IOPS是1000

e、针对数据库的DR有两种方式，一个是mirror everything,一个是mirror log files only

f、ERM保留一个端口，ISCSI端口不支持ERM

g、flashcopy和ERM可以单独购买，而volumecopy必须是和flashcopy捆绑在一起

h、 The DS3500 can mirror to another DS3500 or another member of the IBM Midrange DS Storage System family running 7.70.xx code or higher.

（1）、如果primary lun是controller A拥有，则secondary lun也是controller A拥有。如果primary lun的owner改变，则secondary lun的owner也会改变。lun的secondary controller ownership 不能通过GUI或主机来改变，只能由primary side来决定。

（2）、如果primary lun有不可读的块，则不能创建mirror 关系；再者，如果在mirroring过程中，发现有了不可读块，mirror relationship 也会fail。

（3）、当由于primary 和 secondary 之间的链路断开或lun 错误阻止了两者之间的通信，primary controller 会置mirrored pair为unsynchronized状态，给主机发IO completed。主机能继续发写请求给primary lun，但对secondary lun的写请求不会发生。这个请求会存在primary site的remote mirror repository中。当两者之间的通信恢复，继续resynchronized

（4）、unsynchronized 和 suspend 状态是不同的，第一个是由于某种条件错误，而第二个是管理员设置的，这两个状态的resynchronization是不同的。suspend 状态需要管理员resume，而unsynchronized状态可以手动或自动的synchronization。

在unsynchronized 或suspend 状态下，只有改变的block 会被复制到secondary lun。

正常情况下，当resume relationship或re-establishing 时，只有改变的block会被发送到remote site。然而在某些情况下，需要primary 和 secondary lun的full synchronization。

如果在interruption期间，primary repository lun的所有数据块都改变了，则需要full sychronization

primary 和 secondary 的任何通信disruption，当再次resynchronization时，都可能导致secondary lun混合新的和旧的数据，这会导致在DR解决方案中数据不可用。

（5）、如果suspend 一个write consistency group里的一个成员，则group里成员的所有镜像关系都被suspend。如果resume了，ERM会尝试resume group里的所有成员。

（6）、如果FC link 故障，在同步和异步模式下的处理情况：

A、同步

primary storage 返回给主机IO complete，主机能继续写IO。当FC link resume时，primary storage 会利用自身记录的delta log bitmap来rewrite secondary storage，来保证数据一致性。

primary 和 secondary storage 之间的通信故障不会导致控制器尝试改变lun ownership的改变。ownership只有在主机path failover时才会改变。

B、异步

如果FC link 中断，primary controller会周期性地重连secondary controller。只要queued requests未超过128，mirrored state就还是synchronized。

一旦FC link resume，会开始resynchronization。

（7）、primary/secondary storage 故障后的处理情况：

A、如果secondary lun故障，也不会影响主机IO，等到secondary lun恢复，resynchronization 进程继续。

B、如果primary controller 故障，则会将lun切换到另一个controller。secondary lun也会切换。

C、如果reset primary controller，那就没时间改变 lun的ownership。重启后，controller 会读取存储在mirror repository lun中的log信息，并用来copy受到影响的数据到secondary lun。强烈建议在reset controller之前暂停mirror 关系。

D、如果secondary controller failure，处理方式和secondary lun failure一样。

（8）、role reversal --将主site和secondary site调换，这一般用于主site故障或maintenance时。

role reversal的步骤：

1. Wait for processing on the host server to complete.

2. Flush all I/O from host server to system storage.

3. Ensure that all I/O requests have been recorded on secondary (no I/O in queue or flight).

4. Change replication direction

5. Verify fabric configuration (the right ports on the right fabric) for host connections to secondary.

6. Restart host application using the secondary logical drive as primary.

7. Confirm application data integrity on the secondary.

8. Repeat these steps to reverse replication direction to original primary.

可以使用SM 的performance monitor来直到primary和secondary上的IO操作何时结束，能安全reverse 角色。

帮老婆的淘宝店做个广告--

链接--http://item.taobao.com/item.htm?spm=a1z10.1.w4004-1197388427.20.kuPXWA&id=35649225275