一则奇怪的案例处理：ORA-00257: archiver error. Connect internal only, until freed

前天，业务反应数据库不能连接

在操作系统通过字符串尝试登陆数据库报：ORA-00257: archiver error. Connect internal only, until freed

解决思路：

1.操作系统清理归档

2.rman清理expired归档

遇到日志不能切换，且归档目录未满的情况，且数据库不能正常关闭的解决思路：

1.查看log group 状态，如果处于inactive状态但是报需要归档的错误

2.强制clear未归档的日志

3.删除clear的日志组，并重建

4.如果还不能switch logfile，重启数据库

详细步骤如下：

查看alert日志：

***********************************************************************



Fatal NI connect error 12170.



  VERSION INFORMATION:

        TNS for HPUX: Version 11.2.0.3.0 - Production

        Oracle Bequeath NT Protocol Adapter for HPUX: Version 11.2.0.3.0 - Production

        TCP/IP NT Protocol Adapter for HPUX: Version 11.2.0.3.0 - Production

  Time: 17-MAR-2015 17:28:08

  Tracing not turned on.

  Tns error struct:

    ns main err code: 12535



TNS-12535: TNS:operation timed out

    ns secondary err code: 12606

    nt main err code: 0

    nt secondary err code: 0

    nt OS err code: 0

  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=132.33.128.28)(PORT=4035))

WARNING: inbound connection timed out (ORA-3136)

 哦，原来是归档满了

1.数据库查询归档位置：

SQL> archive log list

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            USE_DB_RECOVERY_FILE_DEST

Oldest online log sequence     5

Next log sequence to archive   5

Current log sequence           7

SQL> show parameter db_recovery_file_dest



NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_recovery_file_dest                string      /Oracle/apps/oracle/recovery

db_recovery_file_dest_size           big integer 4977M



SQL> select * from v$recovery_file_dest



NAME                                               SPACE_LIMIT SPACE_USED SPACE_RECLAIMABLE NUMBER_OF_FILES

-------------------------------------------------- ----------- ---------- ----------------- ---------------

/Oracle/apps/oracle/recovery                        5218762752 4405968896                 0               3

2.进入操作系统，手动删除所有归档

3.rman检查并清理归档

RMAN> crosscheck archivelog all;



using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=235 device type=DISK

validation failed for archived log

archived log file name=/Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_03/o1_mf_1_2_bf1o3ohr_.arc RECID=1 STAMP=870732036

validation failed for archived log

archived log file name=/Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_12/o1_mf_1_3_bfqxh0sz_.arc RECID=2 STAMP=871461477

validation failed for archived log

archived log file name=/Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_19/o1_mf_1_4_bgcv464n_.arc RECID=3 STAMP=872114452

Crosschecked 3 objects



RMAN> list expired archivelog all;



List of Archived Log Copies for database with db_unique_name GJJS

=====================================================================



Key     Thrd Seq     S Low Time 

------- ---- ------- - ---------

1       1    2       X 28-JAN-15

        Name: /Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_03/o1_mf_1_2_bf1o3ohr_.arc



2       1    3       X 03-FEB-15

        Name: /Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_12/o1_mf_1_3_bfqxh0sz_.arc



3       1    4       X 12-FEB-15

        Name: /Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_19/o1_mf_1_4_bgcv464n_.arc





RMAN> delete expired archivelog all;



released channel: ORA_DISK_1

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=235 device type=DISK

List of Archived Log Copies for database with db_unique_name GJJS

=====================================================================



Key     Thrd Seq     S Low Time 

------- ---- ------- - ---------

1       1    2       X 28-JAN-15

        Name: /Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_03/o1_mf_1_2_bf1o3ohr_.arc



2       1    3       X 03-FEB-15

        Name: /Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_12/o1_mf_1_3_bfqxh0sz_.arc



3       1    4       X 12-FEB-15

        Name: /Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_19/o1_mf_1_4_bgcv464n_.arc





Do you really want to delete the above objects (enter YES or NO)? yes

deleted archived log

archived log file name=/Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_03/o1_mf_1_2_bf1o3ohr_.arc RECID=1 STAMP=870732036

deleted archived log

archived log file name=/Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_12/o1_mf_1_3_bfqxh0sz_.arc RECID=2 STAMP=871461477

deleted archived log

archived log file name=/Oracle/apps/oracle/recovery/GJJS/archivelog/2015_02_19/o1_mf_1_4_bgcv464n_.arc RECID=3 STAMP=872114452

Deleted 3 EXPIRED objects

 4.改变归档目录，是归档不放入闪回目录中

alter system set log_archive_dest_1='location=/Oracle/apps/oracle/arch';

 OK

到这里，归档空间已经清理完成，但是在操作系统上再次测试的时候依然报ORA-00257: archiver error. Connect internal only, until freed，很奇怪~~

尝试数据库中切换日志

SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          7 INACTIVE

         2          8 CURRENT

         3          6 INACTIVE

SQL>alter system switch logfile;   





----夯住不动，alert日志中也未出现任何错误

希望通过重启数据库解决：

但是执行shutdown immediate后，数据库无反应，看alert日志

License high water mark = 45

Stopping Job queue slave processes, flags = 7

Tue Mar 17 16:42:09 2015

Errors in file /Oracle/apps/oracle/diag/rdbms/gjjs/gjjs/trace/gjjs_j002_28185.trc:

ORA-12012: error on auto execute of job "EXFSYS"."RLM$EVTCLEANUP"

ORA-01089: immediate shutdown in progress - no operations are permitted

ORA-01089: immediate shutdown in progress - no operations are permitted

Process ID:

Session ID: 0 Serial number: 0

Tue Mar 17 16:42:09 2015

Errors in file /Oracle/apps/oracle/diag/rdbms/gjjs/gjjs/trace/gjjs_j001_17723.trc:

ORA-12012: error on auto execute of job "EXFSYS"."RLM$SCHDNEGACTION"

ORA-01089: immediate shutdown in progress - no operations are permitted

ORA-01089: immediate shutdown in progress - no operations are permitted

Process ID:

Session ID: 0 Serial number: 0

Job queue slave processes stopped

 CTRL+C结束，奇怪~~~

可能是日志文件出现错误，下面进行清理，并重建日志文件

SQL> alter database drop logfile group 3;

alter database drop logfile group 3

*

ERROR at line 1:

ORA-00350: log 3 of instance gjjs (thread 1) needs to be archived

ORA-00312: online log 3 thread 1: '/dev/vg03/rlv_vg03_2g_redo003'





SQL> alter database clear logfile group 3;

alter database clear logfile group 3

*

ERROR at line 1:

ORA-00350: log 3 of instance gjjs (thread 1) needs to be archived

ORA-00312: online log 3 thread 1: '/dev/vg03/rlv_vg03_2g_redo003'





SQL> alter database clear unarchived logfile group 3;



Database altered.



SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          7 INACTIVE

         2          8 CURRENT

         3          0 UNUSED



SQL> select member from v$logfile;



MEMBER

--------------------------------------------------------------------------------

/dev/vg03/rlv_vg03_2g_redo001

/dev/vg03/rlv_vg03_2g_redo002

/dev/vg03/rlv_vg03_2g_redo003



SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          7 INACTIVE

         2          8 CURRENT

         3          0 UNUSED



SQL> alter database drop logfile group 3;



Database altered.



SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          7 INACTIVE

         2          8 CURRENT

SQL> alter database add logfile group 3 '/dev/vg03/rlv_vg03_2g_redo03.log' size 2046M reuse;



Database altered.

 查看，再次切换

SQL> select member from v$logfile;



MEMBER

--------------------------------------------------------------------------------

/dev/vg03/rlv_vg03_2g_redo001

/dev/vg03/rlv_vg03_2g_redo002

/dev/vg03/rlv_vg03_2g_redo03.log



SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          7 INACTIVE

         2          8 CURRENT

         3          0 UNUSED



SQL> alter system switch logfile;



System altered.



SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          7 INACTIVE

         2          8 ACTIVE

         3          9 CURRENT

SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          7 INACTIVE

         2          8 ACTIVE

         3          9 CURRENT



-----group 2一直处于active状态，做一个checkpoint，使日志无效



SQL> alter system checkpoint;



System altered.



SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          7 INACTIVE

         2          8 INACTIVE

         3          9 CURRENT



----切换的时候依然不能切换，重建所有的日志组



SQL> alter database clear logfile group 1;

alter database clear logfile group 1

*

ERROR at line 1:

ORA-00350: log 1 of instance gjjs (thread 1) needs to be archived

ORA-00312: online log 1 thread 1: '/dev/vg03/rlv_vg03_2g_redo001'





SQL> alter database clear unarchived logfile group 1;



Database altered.



SQL> alter database drop logfile group 1;



Database altered.



SQL> alter database add logfile group 1 '/dev/vg03/rlv_vg03_2g_redo01.log' size 2046M reuse;



Database altered.



SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          0 UNUSED

         2          8 INACTIVE

         3          9 CURRENT



SQL> alter database clear unarchived logfile group 2;



Database altered.



SQL>  alter database drop logfile group 2;



Database altered.



SQL> alter database add logfile group 2 '/dev/vg03/rlv_vg03_2g_redo02.log' size 2046M reuse;



Database altered.



SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          0 UNUSED

         2          0 UNUSED

         3          9 CURRENT



SQL> alter system switch logfile;



System altered.



SQL> /



System altered.

SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1         10 INACTIVE

         2         11 CURRENT

         3          9 INACTIVE

SQL> alter database clear unarchived logfile group 3;



Database altered.



SQL> c/3/1

  1* alter database clear unarchived logfile group 1

SQL> /



Database altered.



SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          0 UNUSED

         2         11 CURRENT

         3          0 UNUSED



----重建日志组后依然不行，重启数据库看看



SQL> shutdown immediate

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> 

-----数据库可以正常关闭

SQL> 

SQL> startup

ORACLE instance started.



Total System Global Area 2.1913E+11 bytes

Fixed Size                  2195616 bytes

Variable Size            1.0415E+11 bytes

Database Buffers         1.1489E+11 bytes

Redo Buffers               88866816 bytes

Database mounted.

Database opened.

SQL> 

SQL> 

----查看状态并切换日志，数据库正常~

SQL> select group#,sequence#,status from v$log;



    GROUP#  SEQUENCE# STATUS

---------- ---------- ----------------

         1          0 UNUSED

         2         11 CURRENT

         3          0 UNUSED



SQL> alter system switch logfile;



System altered.



SQL> /



System altered.



SQL> /



System altered.



SQL> /



System altered.



SQL> /



System altered.

 难道是修改log_arch_dest_1参数需要重启数据库？？？说不通！！！