- 故障现象:
- 一、错误日志
- Thu Nov 10 17:08:43 2011
- ORACLE Instance zjport - Archival Error
- Thu Nov 10 17:08:43 2011
- ORA-16038: log 1 sequence# 101 cannot be archived
- ORA-19809: limit exceeded for recovery files
- ORA-00312: online log 1 thread 1: '/home/oracle/oradata/online/redo01.log'
- Thu Nov 10 17:08:43 2011
- Errors in file /home/oracle/admin/online/bdump/online_arc1_30097.trc:
- ORA-16038: log 1 sequence# 101 cannot be archived
- ORA-19809: limit exceeded for recovery files
- ORA-00312: online log 1 thread 1: '/home/oracle/oradata/online/redo01.log'
- Thu Nov 10 17:09:43 2011
- ARCH: Archival stopped, error occurred. Will continue retrying
- Thu Nov 10 17:09:43 2011
- ORACLE Instance online - Archival Error
- Thu Nov 10 17:09:43 2011
- ORA-16014: log 1 sequence# 101 not archived, no available destinations
- ORA-00312: online log 1 thread 1: '/home/oracle/oradata/online/redo01.log'
- Thu Nov 10 17:09:43 2011
- Errors in file /home/oracle/admin/online/bdump/online_arc0_30095.trc:
- ORA-16014: log 1 sequence# 101 not archived, no available destinations
- ORA-00312: online log 1 thread 1: '/home/oracle/oradata/online/redo01.log'
- Thu Nov 10 17:14:43 2011
- Errors in file /home/oracle/admin/online/bdump/online_arc1_30097.trc:
- ORA-19815: WARNING: db_recovery_file_dest_size of 2147483648 bytes is 100.00% used, and has 0 remaining bytes available.
- Thu Nov 10 17:14:43 2011
- ************************************************************************
- You have following choices to free up space from flash recovery area:
- 1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
- then consider changing RMAN ARCHIVELOG DELETION POLICY.
- 2. Back up files to tertiary device such as tape using RMAN
- BACKUP RECOVERY AREA command.
- 3. Add disk space and increase db_recovery_file_dest_size parameter to
- reflect the new space.
- 4. Delete unnecessary files using RMAN DELETE command. If an operating
- system command was used to delete files, then use RMAN CROSSCHECK and
- DELETE EXPIRED commands.
- ************************************************************************
- 二、日志切换失败
- SQL> alter system switch logfile;
- alter system switch logfile
- 长时间无响应,后台日志提示
- Thu Nov 10 17:14:43 2011
- ORACLE Instance online - Archival Error
- Thu Nov 10 17:14:43 2011
- ORA-16038: log 1 sequence# 101 cannot be archived
- ORA-19809: limit exceeded for recovery files
- ORA-00312: online log 1 thread 1: '/home/oracle/oradata/online/redo01.log'
- Thu Nov 10 17:14:43 2011
- Errors in file /home/oracle/admin/online/bdump/online_arc1_30097.trc:
- ORA-16038: log 1 sequence# 101 cannot be archived
- ORA-19809: limit exceeded for recovery files
- ORA-00312: online log 1 thread 1: '/home/oracle/oradata/online/redo01.log'
- Thu Nov 10 17:15:43 2011
- ARCH: Archival stopped, error occurred. Will continue retrying
- Thu Nov 10 17:15:43 2011
- ORACLE Instance online - Archival Error
- Thu Nov 10 17:15:43 2011
- ORA-16014: log 1 sequence# 101 not archived, no available destinations
- ORA-00312: online log 1 thread 1: '/home/oracle/oradata/online/redo01.log'
- Thu Nov 10 17:15:43 2011
- Errors in file /home/oracle/admin/online/bdump/online_arc0_30095.trc:
- ORA-16014: log 1 sequence# 101 not archived, no available destinations
- ORA-00312: online log 1 thread 1: '/home/oracle/oradata/online/redo01.log'
原因分析:
ORACLE log_archive_dest_1 未指定导致归档到flash_recovery_area路径,这点从后文的RMAN备份对归档部分的处理日志可以明显看出来,例如其中一个归档路径为“home/oracle/flash_recovery_area/online/archivelog/2011_10_13/o1_mf_1_52_79ftss3d_.arc”
- 解决方法:
- 方法一:指定log_archive_dest_1到指定目录(此为永久解决方法,但是需要重启)
- --归档路径
- SQL> alter system set log_archive_dest_1='LOCATION=/u01/oracle/archivelog';
- --归档命名格式
- SQL> alter system set log_archive_max_processes = 5;
- SQL> alter system set log_archive_format = "archive_%t_%s_%r.arc" scope=spfile;
- 方法二:increase db_recovery_file_dest_size parameter to reflect the new space
- 这种方法需要重启数据库才能使配置生效,在生产系统中显然不适合
- SQL> show parameter db_recovery_file_dest_size;
- NAME TYPE VALUE
- ------------------------------------ ----------- ------------------------------
- db_recovery_file_dest_size big integer 2G
- SQL> alter system set db_recovery_file_dest_size=4G scope=spfile;
- System altered.
- 方法三:RMAN备份后删除归档,清空归档占用空间,对生产无影响(此为快速恢复数据库的方法)
- [oracle@L-DB-128-46 ~]$ rman target /
- Recovery Manager: Release 10.2.0.4.0 - Production on 星期四 11月 10 17:11:10 2011
- Copyright (c) 1982, 2007, Oracle. All rights reserved.
- connected to target database: online (DBID=2711954858)
- RMAN> run {
- allocate channel 'c1' type disk format '/home/oracle/backup/full_db_20111110.bak';
- backup database plus archivelog delete all input;
- release channel c1;
- }2> 3> 4> 5>
- using target database control file instead of recovery catalog
- allocated channel: c1
- channel c1: sid=145 devtype=DISK
- Starting backup at 2011-11-10 17:17:31
- channel c1: starting archive log backupset
- channel c1: specifying archive log(s) in backup set
- input archive log thread=1 sequence=52 recid=51 stamp=764459065
- input archive log thread=1 sequence=53 recid=53 stamp=764459065
- input archive log thread=1 sequence=54 recid=52 stamp=764459065
- input archive log thread=1 sequence=55 recid=54 stamp=764459071
- input archive log thread=1 sequence=56 recid=55 stamp=764459076
- input archive log thread=1 sequence=57 recid=56 stamp=764492454
- input archive log thread=1 sequence=58 recid=57 stamp=764548216
- input archive log thread=1 sequence=59 recid=58 stamp=764575211
- input archive log thread=1 sequence=60 recid=59 stamp=764640050
- input archive log thread=1 sequence=61 recid=60 stamp=764720252
- input archive log thread=1 sequence=62 recid=61 stamp=764803366
- input archive log thread=1 sequence=63 recid=62 stamp=764845224
- input archive log thread=1 sequence=64 recid=63 stamp=764892556
- input archive log thread=1 sequence=65 recid=64 stamp=764972503
- input archive log thread=1 sequence=66 recid=65 stamp=765015358
- input archive log thread=1 sequence=67 recid=66 stamp=765064813
- input archive log thread=1 sequence=68 recid=67 stamp=765132696
- input archive log thread=1 sequence=69 recid=68 stamp=765152438
- input archive log thread=1 sequence=70 recid=69 stamp=765208861
- input archive log thread=1 sequence=71 recid=70 stamp=765270027
- input archive log thread=1 sequence=72 recid=71 stamp=765329990
- input archive log thread=1 sequence=73 recid=72 stamp=765410404
- input archive log thread=1 sequence=74 recid=73 stamp=765441266
- input archive log thread=1 sequence=75 recid=74 stamp=765496803
- input archive log thread=1 sequence=76 recid=75 stamp=765530600
- input archive log thread=1 sequence=77 recid=76 stamp=765583207
- input archive log thread=1 sequence=78 recid=77 stamp=765622706
- input archive log thread=1 sequence=79 recid=78 stamp=765669611
- input archive log thread=1 sequence=80 recid=79 stamp=765727244
- input archive log thread=1 sequence=81 recid=80 stamp=765764533
- input archive log thread=1 sequence=82 recid=81 stamp=765820824
- input archive log thread=1 sequence=83 recid=82 stamp=765882059
- input archive log thread=1 sequence=84 recid=83 stamp=765962785
- input archive log thread=1 sequence=85 recid=84 stamp=766015206
- input archive log thread=1 sequence=86 recid=85 stamp=766049222
- input archive log thread=1 sequence=87 recid=86 stamp=766101611
- input archive log thread=1 sequence=88 recid=87 stamp=766148505
- input archive log thread=1 sequence=89 recid=88 stamp=766189244
- input archive log thread=1 sequence=90 recid=89 stamp=766251727
- input archive log thread=1 sequence=91 recid=90 stamp=766289777
- input archive log thread=1 sequence=92 recid=91 stamp=766360805
- input archive log thread=1 sequence=93 recid=92 stamp=766383429
- input archive log thread=1 sequence=94 recid=93 stamp=766423722
- input archive log thread=1 sequence=95 recid=94 stamp=766486811
- input archive log thread=1 sequence=96 recid=95 stamp=766548055
- input archive log thread=1 sequence=97 recid=96 stamp=766620005
- input archive log thread=1 sequence=98 recid=97 stamp=766645228
- input archive log thread=1 sequence=99 recid=98 stamp=766706408
- input archive log thread=1 sequence=100 recid=99 stamp=766749605
- channel c1: starting piece 1 at 2011-11-10 17:17:32
- channel c1: finished piece 1 at 2011-11-10 17:17:57
- piece handle=/home/oracle/backup/full_db_20111110.bak tag=TAG20111110T171731 comment=NONE
- channel c1: backup set complete, elapsed time: 00:00:26
- channel c1: deleting archive log(s)
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_13/o1_mf_1_52_79ftss3d_.arc recid=51 stamp=764459065
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_13/o1_mf_1_53_79ftss8m_.arc recid=53 stamp=764459065
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_13/o1_mf_1_54_79ftss8t_.arc recid=52 stamp=764459065
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_13/o1_mf_1_55_79ftsyyr_.arc recid=54 stamp=764459071
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_13/o1_mf_1_56_79ftt3tf_.arc recid=55 stamp=764459076
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_14/o1_mf_1_57_79gvf6p0_.arc recid=56 stamp=764492454
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_14/o1_mf_1_58_79jkvqxd_.arc recid=57 stamp=764548216
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_15/o1_mf_1_59_79kd7c5c_.arc recid=58 stamp=764575211
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_16/o1_mf_1_60_79mcklq6_.arc recid=59 stamp=764640050
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_16/o1_mf_1_61_79osvw5g_.arc recid=60 stamp=764720252
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_17/o1_mf_1_62_79rc16n2_.arc recid=61 stamp=764803366
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_18/o1_mf_1_63_79smx8qs_.arc recid=62 stamp=764845224
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_18/o1_mf_1_64_79v24d0y_.arc recid=63 stamp=764892556
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_19/o1_mf_1_65_79xj6pvv_.arc recid=64 stamp=764972503
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_20/o1_mf_1_66_79yt1y2z_.arc recid=65 stamp=765015358
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_20/o1_mf_1_67_7b0bcfbg_.arc recid=66 stamp=765064813
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_21/o1_mf_1_68_7b2dnr9h_.arc recid=67 stamp=765132696
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_21/o1_mf_1_69_7b2zxpbj_.arc recid=68 stamp=765152438
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_22/o1_mf_1_70_7b4q0x6c_.arc recid=69 stamp=765208861
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_23/o1_mf_1_71_7b6lrclk_.arc recid=70 stamp=765270027
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_23/o1_mf_1_72_7b8fb6lm_.arc recid=71 stamp=765329990
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_24/o1_mf_1_73_7bbvv4c6_.arc recid=72 stamp=765410404
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_25/o1_mf_1_74_7bcszloc_.arc recid=73 stamp=765441266
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_25/o1_mf_1_75_7bfj7389_.arc recid=74 stamp=765496803
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_26/o1_mf_1_76_7bgk782h_.arc recid=75 stamp=765530600
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_26/o1_mf_1_77_7bj4m6wd_.arc recid=76 stamp=765583207
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_27/o1_mf_1_78_7bkc5lpk_.arc recid=77 stamp=765622706
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_27/o1_mf_1_79_7blrzcps_.arc recid=78 stamp=765669611
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_28/o1_mf_1_80_7bnk8d5x_.arc recid=79 stamp=765727244
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_29/o1_mf_1_81_7boooo8f_.arc recid=80 stamp=765764533
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_29/o1_mf_1_82_7bqdnrc7_.arc recid=81 stamp=765820824
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_30/o1_mf_1_83_7bs8gbxh_.arc recid=82 stamp=765882059
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_31/o1_mf_1_84_7bvq91kg_.arc recid=83 stamp=765962785
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_10_31/o1_mf_1_85_7bxbh66d_.arc recid=84 stamp=766015206
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_01/o1_mf_1_86_7bycp6nw_.arc recid=85 stamp=766049222
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_01/o1_mf_1_87_7bzyvbv0_.arc recid=86 stamp=766101611
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_02/o1_mf_1_88_7c1dns1p_.arc recid=87 stamp=766148505
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_02/o1_mf_1_89_7c2nfwdp_.arc recid=88 stamp=766189244
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_03/o1_mf_1_90_7c4kgh7y_.arc recid=89 stamp=766251727
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_04/o1_mf_1_91_7c5pmk14_.arc recid=90 stamp=766289777
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_04/o1_mf_1_92_7c7vz5mp_.arc recid=91 stamp=766360805
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_05/o1_mf_1_93_7c8l259j_.arc recid=92 stamp=766383429
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_05/o1_mf_1_94_7c9sfbg5_.arc recid=93 stamp=766423722
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_06/o1_mf_1_95_7ccq0v8h_.arc recid=94 stamp=766486811
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_07/o1_mf_1_96_7cfltpy7_.arc recid=95 stamp=766548055
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_07/o1_mf_1_97_7chs35kp_.arc recid=96 stamp=766620005
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_08/o1_mf_1_98_7cjkqd4b_.arc recid=97 stamp=766645228
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_08/o1_mf_1_99_7clfh7z5_.arc recid=98 stamp=766706408
- archive log filename=/home/oracle/flash_recovery_area/online/archivelog/2011_11_09/o1_mf_1_100_7cmqo581_.arc recid=99 stamp=766749605
- Finished backup at 2011-11-10 17:17:58
- Starting backup at 2011-11-10 17:17:58
- channel c1: starting full datafile backupset
- channel c1: specifying datafile(s) in backupset
- input datafile fno=00005 name=/home/oracle/oradata/online/cuteinfo02.dbf
- input datafile fno=00006 name=/home/oracle/oradata/online/rman_tbs01.dbf
- input datafile fno=00001 name=/home/oracle/oradata/online/system01.dbf
- input datafile fno=00003 name=/home/oracle/oradata/online/sysaux01.dbf
- input datafile fno=00002 name=/home/oracle/oradata/online/undotbs01.dbf
- input datafile fno=00004 name=/home/oracle/oradata/online/users01.dbf
- channel c1: starting piece 1 at 2011-11-10 17:17:58
- released channel: c1
- RMAN-00571: ===========================================================
- RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
- RMAN-00571: ===========================================================
- RMAN-03002: failure of backup plus archivelog command at 11/10/2011 17:17:59------------------此错误是笔者输入了固定的文件名导致,如下改为full_db_%U.bak后输出正常
- ORA-19504: failed to create file "/home/oracle/backup/full_db_20111110.bak"
- ORA-27038: created file already exists
- Additional information: 1
- RMAN> run {
- allocate channel 'c1' type disk format '/home/oracle/backup/full_db_%U.bak';
- backup database plus archivelog delete all input;
- release channel c1;
- }2> 3> 4> 5>
- allocated channel: c1
- channel c1: sid=145 devtype=DISK
- Starting backup at 2011-11-10 17:20:22
- specification does not match any archive log in the recovery catalog
- backup cancelled because all files were skipped
- Finished backup at 2011-11-10 17:20:22
- Starting backup at 2011-11-10 17:20:22
- channel c1: starting full datafile backupset
- channel c1: specifying datafile(s) in backupset
- input datafile fno=00005 name=/home/oracle/oradata/online/cuteinfo02.dbf
- input datafile fno=00006 name=/home/oracle/oradata/online/rman_tbs01.dbf
- input datafile fno=00001 name=/home/oracle/oradata/online/system01.dbf
- input datafile fno=00003 name=/home/oracle/oradata/online/sysaux01.dbf
- input datafile fno=00002 name=/home/oracle/oradata/online/undotbs01.dbf
- input datafile fno=00004 name=/home/oracle/oradata/online/users01.dbf
- channel c1: starting piece 1 at 2011-11-10 17:20:22
- channel c1: finished piece 1 at 2011-11-10 17:20:37
- piece handle=/home/oracle/backup/full_db_03mrap2m_1_1.bak tag=TAG20111110T172022 comment=NONE
- channel c1: backup set complete, elapsed time: 00:00:15
- channel c1: starting full datafile backupset
- channel c1: specifying datafile(s) in backupset
- including current control file in backupset
- including current SPFILE in backupset
- channel c1: starting piece 1 at 2011-11-10 17:20:38
- channel c1: finished piece 1 at 2011-11-10 17:20:39
- piece handle=/home/oracle/backup/full_db_04mrap35_1_1.bak tag=TAG20111110T172022 comment=NONE
- channel c1: backup set complete, elapsed time: 00:00:02
- Finished backup at 2011-11-10 17:20:39
- Starting backup at 2011-11-10 17:20:39
- specification does not match any archive log in the recovery catalog
- backup cancelled because all files were skipped
- Finished backup at 2011-11-10 17:20:39
- released channel: c1
- RMAN>
故障解除确认:
尝试切换归档成功:
SQL> alter system switch logfile;
System altered.
SQL>
后台日志如下:
Thu Nov 10 17:20:44 2011
db_recovery_file_dest_size of 2048 MB is 2.02% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
kcrrdmx: Successful archiving of previously failed ORL
Archiver process freed from errors. No longer stopped
Thu Nov 10 17:21:33 2011
Thread 1 advanced to log sequence 104 (LGWR switch)
Current log# 1 seq# 104 mem# 0: /home/oracle/oradata/online/redo01.log
后续解决方法:
修改初试化参数,使归档到相应的磁盘路径
--归档路径
SQL> alter system set log_archive_dest_1='LOCATION=/u01/oracle/archivelog';
--归档命名格式
SQL> alter system set log_archive_max_processes = 5;
SQL> alter system set log_archive_format = "archive_%t_%s_%r.arc" scope=spfile;