TimesTen 数据库复制学习：5. 设定复制网络传输的return service

return service指在复制源和目标之间的同步模式，可以是no return(异步)，return receipt(准同步)和return twosafe(全同步)。
本文的描述对于active standby和classic replication都适用。

脚本准备

$ cat insert1.sql
INSERT INTO employees VALUES
( 202,
‘Pat’,
‘Fay’,
‘PFAY’,
‘603-123-7777’,
TO_DATE(‘17-AUG-1997’, ‘dd-MON-yyyy’),
‘MK_REP’,
6000,
NULL,
201,
20
);

$ cat insert2.sql
INSERT INTO employees VALUES
( 203,
‘Judy’,
‘Fox’,
‘JFOX’,
‘603-123-7777’,
TO_DATE(‘17-AUG-1997’, ‘dd-MON-yyyy’),
‘MK_REP’,
6000,
NULL,
201,
20
);

$ cat stoprep.sql
call ttrepstop;
drop active standby pair;

为不同的复制目标使用不同的return service

Timesten支持一对多复制，对于复制的目标数据库(对于ASP，目标是standby master或subscriber，)，可以指定不同的复制策略，具体语法参见”CREATE ACTIVE STANDBY PAIR” and “CREATE REPLICATION” (classic复制)，例如：

CREATE REPLICATION Owner.SchemeName ELEMENT ElementNameElementType MASTER DatabaseName ON "HostName" SUBSCRIBER SubDatabase1 ON "HostName" no return SUBSCRIBER SubDatabase2 ON "HostName" return receipt;

以下表示使用相同的return service:

CREATE REPLICATION Owner.SchemeName ELEMENT ElementNameElementType MASTER DatabaseName ON "HostName" SUBSCRIBER SubDatabase1 ON "HostName", SubDatabase2 ON "HostName" return twosafe;

RETURN RECEIPT

return receipt的行为特征如下：

With return receipt enabled, when your application commits a transaction for an element on the master database, the application remains blocked until the subscriber/standby acknowledges receipt of the transaction update. If the master is replicating the element to multiple subscribers, the application remains blocked until all of the subscribers have acknowledged receipt of the transaction update.

If the standby or subscriber is unable to acknowledge receipt of the transaction within a configurable timeout period, your application receives a tt_ErrRepReturnFailed (8170) warning on its commit request.

以下是一个classic 复制的例子：

CREATE REPLICATION tablerep ELEMENT e TABLE employees MASTER master1 SUBSCRIBER subscriber1 RETURN RECEIPT;

如果由于网络原因或由于复制代理未启动，导致目标不可达，那么active master也不会无限等待下去，而是可以设置超时，缺省为10秒，超时后产生警告，但交易在active master端仍是成功的，待通讯恢复后，数据仍会同步到对方。
例如我们停止目标端的复制代理来模拟通讯故障，然后在主点插入数据，这时只是会产生警告，但数据插入是成功的：

Command> @insert1
Warning  8170: Receipt or commit acknowledgement not returned in the specified timeout interval for XID:1.85
1 row inserted. <- 1行数据成功插入

RETURN RECEIPT BY REQUEST

RETURN RECEIPT 对于所有的交易都会要求receipt（回执），或者说都会打开return service服务。而RETURN RECEIPT BY REQUEST是一种混合模式，在缺省时，return service关闭，相当于no return，而当用ttRepSyncSet针对指定的交易打开return service后，就又会要求receipt。

BY REQUEST可以理解成ON DEMAND，MANUALLY ENABLED RETURN SERVICE。

CREATE TABLE employees
  ( employee_id NUMBER(6) PRIMARY KEY,
  first_name VARCHAR2(20),
  last_name VARCHAR2(25) NOT NULL,
  email VARCHAR2(25) NOT NULL UNIQUE,
  phone_number VARCHAR2(20),
  hire_date DATE NOT NULL,
  job_id VARCHAR2(10) NOT NULL,
  salary NUMBER(8,2),
  commission_pct NUMBER(2,2),
  manager_id NUMBER(6),
  department_id NUMBER(4)
  ) ; 

CREATE REPLICATION tablerep
  ELEMENT e TABLE employees
  MASTER master1
  SUBSCRIBER subscriber1
  RETURN RECEIPT BY REQUEST;

Command> call ttrepsyncget;
< 00, 10, 1 >
1 row found.
Command> repschemes;

Replication Scheme ORACLE.TABLEREP:

  Element: E                              
  Type: Table ORACLE.EMPLOYEES
  Master Store: MASTER1 on TIMESTEN-HOL Transmit Durable
  Subscriber Store: SUBSCRIBER1 on TIMESTEN-HOL Return Receipt By Request

  Store: MASTER1 on TIMESTEN-HOL
    Port: (auto)
    Log Fail Threshold: (none)
    Retry Timeout: 120 seconds
    Compress Traffic: Disabled

  Store: SUBSCRIBER1 on TIMESTEN-HOL
    Port: (auto)
    Log Fail Threshold: (none)
    Retry Timeout: 120 seconds
    Compress Traffic: Disabled
    Return Service Wait Time: 10 seconds
    Return Service on Replication Stop: Disabled
    Return Service Failure Policy: (none)

1 replication scheme found.

先不启动standby上的复制代理，只在active master上启动复制代理，并插入数据。

call ttrepstart;

@insert1

1 row inserted.

居然成功了，因为此时return service是关闭的。然后主动的要求return receipt。

autocommit off;
CALL ttRepSyncSet(0x01, 5, 1); <- 0x01表示打开return service；5表示超时；1只对return twosafe有意义，此处无意义。
@insert2
1 row inserted. <- 在active master中插入成功了，但还未复制到standby
Command> commit; <- 5秒后超时，因为standby上的复制代理未启动。commit后，return service又重新关闭。
Warning  8170: Receipt or commit acknowledgement not returned in the specified timeout interval for XID:1.123

Command> select * from employees;
< 202, Pat, Fay, PFAY, 603-123-7777, 1997-08-17 00:00:00, MK_REP, 6000, <NULL>, 201, 20 >
< 203, Judy, Fox, JFOX, 603-123-7777, 1997-08-17 00:00:00, MK_REP, 6000, <NULL>, 201, 20 >
2 rows found. <- 此时standby上还没有数据

在standby上启动复制代理，数据全部同步到standby

Command> select * from employees;
0 rows found.
Command> call ttrepstart;
Command> select * from employees;
< 202, Pat, Fay, PFAY, 603-123-7777, 1997-08-17 00:00:00, MK_REP, 6000, <NULL>, 201, 20 >
< 203, Judy, Fox, JFOX, 603-123-7777, 1997-08-17 00:00:00, MK_REP, 6000, <NULL>, 201, 20 >
2 rows found.

补充一句，truncate table也是可以复制的。

RETURN TWOSAFE

The return twosafe service ensures that each replicated transaction is committed on the standby database before it is committed on the active database. If replication is unable to verify the transaction has been committed on the standby or subscriber, it returns notification of the error. Upon receiving an error, the application can either take a unique action or fall back on preconfigured actions, depending on the type of failure.

RETURN TWOSAFE是DataStore的属性，不能为某个表指定此属性。例如：

Command> CREATE REPLICATION tablerep
       >   ELEMENT e TABLE employees
       >   MASTER master1
       >   SUBSCRIBER subscriber1
       >   RETURN TWOSAFE;
 8193: The TWOSAFE subscriber attribute is only permitted on a DATASTORE element.
The command failed.

CREATE ACTIVE STANDBY PAIR 
 master1, master2 
 RETURN TWOSAFE;

另外，指定RETURN TWOSAFE后，autocommit必须设置为关闭，否则报错：

8099: TWOSAFE operation not permitted with AutoCommit = 1.

交易必须主动的commit。
RETURN TWOSAFE也属于return service的一种，因此也有超时，当目标库无法通讯时，不影响master库端的数据更新，当通讯恢复后，数据自动同步。
我们用ttrepstop停止standby的复制代理，模拟通讯失败，然后在master插入数据：

Command> autocommit;
autocommit = 0 (OFF) <- 在TWOSAFE模式下，autocommit必须为off
Command> commit;
Command> @insert2
1 row inserted. <- 通讯中断不影响主库的数据插入
Command> commit; <- 等待10秒然后timeout
 8170: Receipt or commit acknowledgement not returned in the specified timeout interval for XID:1.149
The command failed.
Command> select * from employees;
< 202, Pat, Fay, PFAY, 603-123-7777, 1997-08-17 00:00:00, MK_REP, 6000, <NULL>, 201, 20 >
< 203, Judy, Fox, JFOX, 603-123-7777, 1997-08-17 00:00:00, MK_REP, 6000, <NULL>, 201, 20 >
2 rows found.

这时仍处于未提交的状态，另起会话登录，查询到记录只有1条
再次commit还是不会成功，因此只有退出，不过退出时，居然在本地提交了，也就是ttrepsync可设置的行为模式
Command> exit
Rolling back active transaction...
 8197: Operation not permitted once commit processing has begun
Committing active transaction...
 8170: Receipt or commit acknowledgement not returned in the specified timeout interval for XID:1.27
Disconnecting...
25000: Invalid transaction state
S1010: Function sequence error
S1010: Function sequence error
Done.

更神奇的是，即使在交易未提交时，执行kill -9, 记录仍然在本地commit了。看来signal 9也被trap了。
这时启动master2上的复制代理，记录就同步了。

启动standby的复制代理，然后数据全部同步：

Command> call ttrepstop;
Command> select * from employees;
0 rows found.
Command> call ttrepstart;
Command> select * from employees;
< 202, Pat, Fay, PFAY, 603-123-7777, 1997-08-17 00:00:00, MK_REP, 6000, <NULL>, 201, 20 >
< 203, Judy, Fox, JFOX, 603-123-7777, 1997-08-17 00:00:00, MK_REP, 6000, <NULL>, 201, 20 >
2 rows found.

RETURN TWOSAFE BY REQUEST

和RETURN RECEIPT BY REQUEST类似，RETURN TWOSAFE BY REQUEST就是按需打开RETURN TWOSAFE的模式。在没有打开时，即为异步(no return)模式。

master1>
CREATE ACTIVE STANDBY PAIR 
 master1, master2 
 RETURN TWOSAFE BY REQUEST;

call ttrepstart;
call ttrepstateset('active'); <- 这一步必须做，否则会报下面的错：
TT12048: Error performing backup at source.  More information can be found in the source's message log
TT8144: [1105783104, 0, noAwt] MASTER1:receiver.c(6735): TT8144: Duplicate not permitted.  Reason: Attempted duplicate from a non-ACTIVE Master to Master. Duplicate is only permitted from the ACTIVE store to the STANDBY store, from the STANDBY store to a SUBSCRIBER, or from the ACTIVE store to a SUBSCRIBER if the STANDBY store has failed
TT16025: [1105783104, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'RECEIVER' (context 0x4455600) starting


$ ttRepAdmin -duplicate -from master1 -host $(hostname) -uid repadmin -pwd timesten master2
此时先不启动复制代理，模拟通讯中断。这时相当于异步模式，即return service没有打开
master1> @insert1
1 row inserted. <- 没有阻塞
Command> commit; <- 没有阻塞

Command> autocommit 0
Command> CALL ttRepSyncSet(0x01, 5, 2); <- 第三个参数是专为TWOSAFE设置的模式，如果设置为1，行为模式在RETURN SERVICE一节已描述，不再赘述
如果等于2，表示目标不可达时，本地提交。当目标可达时，未同步的记录会同步。
Command> call ttrepsyncget;
< 01, 5, 2 >

@insert2
1 row inserted.
Command> commit; <- 阻塞，超时返回
 8170: Receipt or commit acknowledgement not returned in the specified timeout interval for XID:1.52
The command failed.

NO RETURN

缺省模式，异步。

master1> CREATE ACTIVE STANDBY PAIR master1, master2 ;
master1> call ttrepstart;
master1> call ttrepstateset('active');
Command> call ttrepstateget;
< ACTIVE, NO GRID >
1 row found.

$ ttRepAdmin -duplicate -from master1 -host $(hostname) -uid repadmin -pwd timesten master2
master2> call ttrepstart;
master2> call ttrepstateget;
< STANDBY, NO GRID >

master1> @insert1;
1 row inserted.

master2> select * from employees;
< 202, Pat, Fay, PFAY, 603-123-7777, 1997-08-17 00:00:00, MK_REP, 6000, <NULL>, 201, 20 >
1 row found.

master2> call ttrepstop;

master1> @insert2
1 row inserted. <- 此处没有任何阻塞
master1> select count(*) from employees; < 2 > 1 row found. master2> select count(*) from employees;
< 1 >
1 row found.
master2> call ttrepstart;
master2> select count(*) from employees; < 2 > 1 row found. 

总结

• TimesTen的复制传输模式支持NO RETURN, RETURN RECEIPT和RETURN TWOSAFE三种模式，分别表示异步，准同步和全同步
• RETURN RECEIPT表示对方接收到记录，RETURN TWOSAFE表示对方接收到记录并提交
• 当复制目标不可达时，缺省会超时10秒，对于RETURN RECEIPT，本地会提交；对于RETURN TWOSAFE，可以设置本地提交或不提交
• ttRepSyncSet和ttRepSyncGet可以设置超时，打开或关闭return service，为TWOSAFE设置当目标不可达时，本地交易是提交还是不提交
• 对于classic复制，TWOSAFE只能用于双向复制, 如下错误提示：

Command> CREATE REPLICATION dsrep
        ELEMENT e DATASTORE 
          MASTER master1
          SUBSCRIBER master2
          RETURN TWOSAFE;
 8195: The TWOSAFE subscriber attribute requires an exclusive bi-directional configuration.
The command failed.