RMAN failed becaue of ORA-19502.

RMAN failed because the following errors from rman log.

channel ORA_DISK_2: starting piece 1 at DEC-28-2010 12:15:14
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 12/28/2010 12:23:39
ORA-19502: write error on file "/global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_oom0mi2h_1_1.bak", blockno 4403201 (blocksize=512
)
ORA-27063: skgfospo: number of bytes read/written is incorrect
Additional information: 597504
Additional information: 1048576

It is because there is no diskspace to hold the backup pieces.

we have set the retention window for two days.

RMAN> show all;

using target database controlfile instead of recovery catalog
RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 2 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO DISK;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_ctrl_%F.bak';
CONFIGURE DEVICE TYPE DISK PARALLELISM 2;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_%U.bak' MAXPIECESIZE 8000 M;
CONFIGURE MAXSETSIZE TO UNLIMITED;

It means the diskspace should be able hold at least three days backup pieces.

use the df -h to see how much disk usage for the two days backups.

The value is 78% for the backup filesystem.

This is the rootcause of the failure.

Let us see the details:

RMAN> list expired backup;

RMAN> report need backup redundancy =2;

Report of files with less than 2 redundant backups
File #bkps Name
---- ----- -----------------------------------------------------

RMAN> report need backup redundancy =1;

Report of files with less than 1 redundant backups
File #bkps Name
---- ----- -----------------------------------------------------

RMAN> report unrecoverable;

Report of files that need backup due to unrecoverable operations
File Type of Backup Required Name
---- ----------------------- -----------------------------------

RMAN> report obsolete;

RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 2 days
no obsolete backups found
RMAN> Crosscheck backup;

allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=770 devtype=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: sid=820 devtype=DISK
。。。。。。。。。。。。。。。。。。。。。
Crosschecked 8 objects?

RMAN> Crosscheck archivelog all;

released channel: ORA_DISK_1
released channel: ORA_DISK_2
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=770 devtype=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: sid=820 devtype=DISK
........................................
Crosschecked 28 objects
RMAN> list expired backup;
List of Backup Sets
===================

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15090 Full 4G DISK 00:17:48 26-DEC-10
BP Key: 15090 Status: EXPIRED Tag: TAG20101226T001503
Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_ntm0fv47_1_1.bak
List of Datafiles in backup set 15090
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
1 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/system01.dbf
4 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/tools01.dbf
6 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/xdb01.dbf
7 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/am_data01.dbf
10 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/cp_inventory_data01.dbf
13 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/pm_data01.dbf
18 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/account_data01.dbf
21 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/wiki01.dbf
23 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/cp_inventory_data02.dbf
24 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/tis_data01.dbf
25 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/xdb02.dbf
26 Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/tools02.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15091 Full 5G DISK 00:22:33 26-DEC-10
BP Key: 15091 Status: EXPIRED Tag: TAG20101226T001503
Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_num0fv48_1_1.bak
List of Datafiles in backup set 15091
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
2 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/undotbs01.dbf
3 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/drsys01.dbf
5 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/users01.dbf
8 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/cta_data01.dbf
9 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/heartbeat_data01.dbf
11 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/jira_data01.dbf
12 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/mast_data01.dbf
14 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/services_data01.dbf
15 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/spyglass_data01.dbf
17 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/pwd_data01.dbf
19 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/hr_data01.dbf
22 Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/smc_mv_logs01.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15092 Full 2G DISK 00:11:18 26-DEC-10
BP Key: 15092 Status: EXPIRED Tag: TAG20101226T001503
Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_nvm0g05s_1_1.bak
List of Datafiles in backup set 15092
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
20 Full 8467277165500 26-DEC-10 /global/oracle2/oradata2/CPPRD/webadmin_data02.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15093 Full 3G DISK 00:12:09 26-DEC-10
BP Key: 15093 Status: EXPIRED Tag: TAG20101226T001503
Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o0m0g0ep_1_1.bak
List of Datafiles in backup set 15093
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
16 Full 8467277516399 26-DEC-10 /global/oracle2/oradata2/CPPRD/webadmin_data01.dbf

BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
15094 2G DISK 00:11:08 26-DEC-10
BP Key: 15094 Status: EXPIRED Tag: TAG20101226T004957
Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o2m0g15n_1_1.bak

List of Archived Logs in backup set 15094
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 157407 8467254043607 25-DEC-10 8467255012759 25-DEC-10
1 157408 8467255012759 25-DEC-10 8467255926649 25-DEC-10
1 157409 8467255926649 25-DEC-10 8467256945794 25-DEC-10
1 157410 8467256945794 25-DEC-10 8467257864568 25-DEC-10
1 157411 8467257864568 25-DEC-10 8467258803216 25-DEC-10
1 157412 8467258803216 25-DEC-10 8467259738992 25-DEC-10
1 157413 8467259738992 25-DEC-10 8467260647589 25-DEC-10
1 157414 8467260647589 25-DEC-10 8467261607633 25-DEC-10
1 157415 8467261607633 25-DEC-10 8467262589366 25-DEC-10
1 157416 8467262589366 25-DEC-10 8467263550238 25-DEC-10
1 157417 8467263550238 25-DEC-10 8467264461024 25-DEC-10
1 157418 8467264461024 25-DEC-10 8467265400438 25-DEC-10
1 157419 8467265400438 25-DEC-10 8467266175528 25-DEC-10
1 157420 8467266175528 25-DEC-10 8467266249095 25-DEC-10
1 157421 8467266249095 25-DEC-10 8467266418398 25-DEC-10
1 157422 8467266418398 25-DEC-10 8467266634369 25-DEC-10
1 157423 8467266634369 25-DEC-10 8467267638357 25-DEC-10
1 157424 8467267638357 25-DEC-10 8467268556163 25-DEC-10
1 157425 8467268556163 25-DEC-10 8467269487701 25-DEC-10
1 157426 8467269487701 25-DEC-10 8467270430555 25-DEC-10
1 157427 8467270430555 25-DEC-10 8467271367767 25-DEC-10
1 157428 8467271367767 25-DEC-10 8467272699523 25-DEC-10
1 157429 8467272699523 25-DEC-10 8467274168819 25-DEC-10
1 157430 8467274168819 25-DEC-10 8467275028116 25-DEC-10
1 157431 8467275028116 25-DEC-10 8467275974031 26-DEC-10
1 157432 8467275974031 26-DEC-10 8467277062422 26-DEC-10
1 157433 8467277062422 26-DEC-10 8467278130963 26-DEC-10
1 157434 8467278130963 26-DEC-10 8467278257188 26-DEC-10

BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
15095 3G DISK 00:13:22 26-DEC-10
BP Key: 15095 Status: EXPIRED Tag: TAG20101226T121506
Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o4m0h9ad_1_1.bak

List of Archived Logs in backup set 15095
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 157446 8467288682614 26-DEC-10 8467289696576 26-DEC-10
1 157447 8467289696576 26-DEC-10 8467290634240 26-DEC-10
1 157448 8467290634240 26-DEC-10 8467291563792 26-DEC-10
1 157449 8467291563792 26-DEC-10 8467292527431 26-DEC-10
1 157450 8467292527431 26-DEC-10 8467293403743 26-DEC-10
1 157451 8467293403743 26-DEC-10 8467294423132 26-DEC-10
1 157452 8467294423132 26-DEC-10 8467295335621 26-DEC-10
1 157453 8467295335621 26-DEC-10 8467296379866 26-DEC-10
1 157454 8467296379866 26-DEC-10 8467297359389 26-DEC-10
1 157455 8467297359389 26-DEC-10 8467298299591 26-DEC-10
1 157456 8467298299591 26-DEC-10 8467299218934 26-DEC-10
1 157457 8467299218934 26-DEC-10 8467300153869 26-DEC-10
1 157458 8467300153869 26-DEC-10 8467301148816 26-DEC-10
1 157459 8467301148816 26-DEC-10 8467302062402 26-DEC-10
1 157460 8467302062402 26-DEC-10 8467303020155 26-DEC-10
1 157461 8467303020155 26-DEC-10 8467303968319 26-DEC-10
1 157462 8467303968319 26-DEC-10 8467304868044 26-DEC-10
1 157463 8467304868044 26-DEC-10 8467305793065 26-DEC-10
1 157464 8467305793065 26-DEC-10 8467306751786 26-DEC-10
1 157465 8467306751786 26-DEC-10 8467307706866 26-DEC-10
1 157466 8467307706866 26-DEC-10 8467308609570 26-DEC-10
1 157467 8467308609570 26-DEC-10 8467309525068 26-DEC-10
1 157468 8467309525068 26-DEC-10 8467310472359 26-DEC-10
1 157469 8467310472359 26-DEC-10 8467311417658 26-DEC-10
1 157470 8467311417658 26-DEC-10 8467312423646 26-DEC-10
1 157471 8467312423646 26-DEC-10 8467313332226 26-DEC-10
1 157472 8467313332226 26-DEC-10 8467314268393 26-DEC-10
1 157473 8467314268393 26-DEC-10 8467315181441 26-DEC-10
1 157474 8467315181441 26-DEC-10 8467316146094 26-DEC-10
1 157475 8467316146094 26-DEC-10 8467317109806 26-DEC-10
1 157476 8467317109806 26-DEC-10 8467318032098 26-DEC-10
1 157477 8467318032098 26-DEC-10 8467319034761 26-DEC-10
1 157478 8467319034761 26-DEC-10 8467319961122 26-DEC-10
1 157479 8467319961122 26-DEC-10 8467320826074 26-DEC-10
1 157480 8467320826074 26-DEC-10 8467321752728 26-DEC-10
1 157481 8467321752728 26-DEC-10 8467322712789 26-DEC-10
1 157482 8467322712789 26-DEC-10 8467323647244 26-DEC-10
1 157483 8467323647244 26-DEC-10 8467324631265 26-DEC-10
1 157484 8467324631265 26-DEC-10 8467324875108 26-DEC-10

BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
15096 3G DISK 00:13:39 26-DEC-10
BP Key: 15096 Status: EXPIRED Tag: TAG20101226T121506
Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o3m0h9ad_1_1.bak

List of Archived Logs in backup set 15096
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 157378 8467226164917 25-DEC-10 8467227143624 25-DEC-10
1 157379 8467227143624 25-DEC-10 8467228142839 25-DEC-10
1 157380 8467228142839 25-DEC-10 8467229106571 25-DEC-10
1 157381 8467229106571 25-DEC-10 8467230041506 25-DEC-10
1 157382 8467230041506 25-DEC-10 8467231042828 25-DEC-10
1 157383 8467231042828 25-DEC-10 8467231985039 25-DEC-10
1 157384 8467231985039 25-DEC-10 8467232954305 25-DEC-10
1 157385 8467232954305 25-DEC-10 8467233913862 25-DEC-10
1 157386 8467233913862 25-DEC-10 8467234866135 25-DEC-10
1 157387 8467234866135 25-DEC-10 8467235811031 25-DEC-10
1 157388 8467235811031 25-DEC-10 8467236720854 25-DEC-10
1 157389 8467236720854 25-DEC-10 8467237680501 25-DEC-10
1 157390 8467237680501 25-DEC-10 8467238649502 25-DEC-10
1 157391 8467238649502 25-DEC-10 8467239555952 25-DEC-10
1 157392 8467239555952 25-DEC-10 8467240548035 25-DEC-10
1 157393 8467240548035 25-DEC-10 8467241457669 25-DEC-10
1 157394 8467241457669 25-DEC-10 8467242451248 25-DEC-10
1 157395 8467242451248 25-DEC-10 8467243407665 25-DEC-10
1 157396 8467243407665 25-DEC-10 8467244349365 25-DEC-10
1 157397 8467244349365 25-DEC-10 8467245248127 25-DEC-10
1 157398 8467245248127 25-DEC-10 8467246246776 25-DEC-10
1 157399 8467246246776 25-DEC-10 8467247347199 25-DEC-10
1 157400 8467247347199 25-DEC-10 8467248252623 25-DEC-10
1 157401 8467248252623 25-DEC-10 8467249257454 25-DEC-10
1 157402 8467249257454 25-DEC-10 8467250187423 25-DEC-10
1 157403 8467250187423 25-DEC-10 8467251166829 25-DEC-10
1 157404 8467251166829 25-DEC-10 8467252147525 25-DEC-10
1 157405 8467252147525 25-DEC-10 8467253093732 25-DEC-10
1 157406 8467253093732 25-DEC-10 8467254043607 25-DEC-10
1 157435 8467278257188 26-DEC-10 8467279175606 26-DEC-10
1 157436 8467279175606 26-DEC-10 8467280093538 26-DEC-10
1 157437 8467280093538 26-DEC-10 8467281078581 26-DEC-10
1 157438 8467281078581 26-DEC-10 8467281995278 26-DEC-10
1 157439 8467281995278 26-DEC-10 8467283023767 26-DEC-10
1 157440 8467283023767 26-DEC-10 8467283929267 26-DEC-10
1 157441 8467283929267 26-DEC-10 8467284919113 26-DEC-10
1 157442 8467284919113 26-DEC-10 8467285935472 26-DEC-10
1 157443 8467285935472 26-DEC-10 8467286842785 26-DEC-10
1 157444 8467286842785 26-DEC-10 8467287756726 26-DEC-10
1 157445 8467287756726 26-DEC-10 8467288682614 26-DEC-10

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15097 Full 2M DISK 00:00:02 26-DEC-10
BP Key: 15097 Status: EXPIRED Tag:
Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_ctrl_c-777430827-20101226-00.bak
SPFILE Included: Modification time: 10-DEC-08

RMAN> report need backup redundancy =1;

Report of files with less than 1 redundant backups
File #bkps Name
---- ----- -----------------------------------------------------
RMAN> report unrecoverable;
Report of files that need backup due to unrecoverable operations
File Type of Backup Required Name
---- ----------------------- -----------------------------------
RMAN> report obsolete;
RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 2 days
Report of obsolete backups and copies
Type Key Completion Time Filename/Handle
-------------------- ------ ------------------ --------------------
Backup Set 15090 26-DEC-10
Backup Piece 15090 26-DEC-10 /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_ntm0fv47_1_1.bak
Backup Set 15091 26-DEC-10
Backup Piece 15091 26-DEC-10 /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_num0fv48_1_1.bak
Backup Set 15092 26-DEC-10
Backup Piece 15092 26-DEC-10 /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_nvm0g05s_1_1.bak
Backup Set 15093 26-DEC-10
Backup Piece 15093 26-DEC-10 /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o0m0g0ep_1_1.bak
Backup Set 15097 26-DEC-10
Backup Piece 15097 26-DEC-10 /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_ctrl_c-777430827-20101226-00.bak
The 26th backup is expired but is not removed correctly.

It take the diskspace then the new backup can not generate because there is no enough diskspace to hold backup pieces.

It sounds simple?

but what cause the 26th backup stay there after being expired?