channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 01/08/2013 14:57:17
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 01/08/2013 14:57:16
ORA-00245: control file backup operation failed
RMAN> show all;
RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET PARALLELISM 2;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK CONNECT '*' FORMAT '+DGRCAT/rmanbak/ORCL2_%U';
CONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT '*';
CONFIGURE CHANNEL 2 DEVICE TYPE DISK CONNECT '*';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/11.2.0/db_2/dbs/snapcf_ORCL1.f'; # default
orarac1poc:ORCL1:/home/oracle> oerr ora 245;
00245, 00000, "control file backup operation failed"
// *Cause: Failed to create a control file backup because some process
// signaled an error during backup creation.
// *Action: Check alert files for further information. This usually happens
// because some process could not access the backup file during
// backup creation. Any process of any instance that starts a read/write
// control file transaction must have an access to the backup control file
// during backup creation.
and from alert.log, below scared me at first glance.
Tue Jan 08 14:57:15 2013
Control file backup creation failed.
Backup target file size found to be zero.
Errors in file /u01/app/oracle/diag/rdbms/orcl/ORCL1/trace/ORCL1_ora_4942.trc:
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Trace file /u01/app/oracle/diag/rdbms/orcl/ORCL1/trace/ORCL1_ora_4942.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_2
System name: Linux
Node name: orarac1poc
Release: 2.6.18-238.el5
Version: #1 SMP Sun Dec 19 14:22:44 EST 2010
Machine: x86_64
VM name: VMWare Version: 6
Instance name: ORCL1
Redo thread mounted by this instance: 1
Oracle process number: 61
Unix process pid: 4942, image: oracle@orarac1poc
*** 2013-01-08 14:57:15.511
*** SESSION ID:(155.15721) 2013-01-08 14:57:15.511
*** CLIENT ID:() 2013-01-08 14:57:15.511
*** SERVICE NAME:(ORCL) 2013-01-08 14:57:15.511
*** MODULE NAME:(backup full datafile) 2013-01-08 14:57:15.511
*** ACTION NAME:(0000146 STARTED16) 2013-01-08 14:57:15.511
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
so, I verified control is there and try to restart this RAC database using below commands without any error shown in alert.log
srvctl start database -d ORCL
srvctl stop database -d ORCL
then, I started google, from below post, I found the answer which is exactly same with my env.
RMAN-03009 and ORA-00245 on 11g R2 RAC
"
As described in MOS Doc [ID 1268725.1] starting from 11.2.0.2, RMAN
fails on RAC when doing controlfile autobackup if RMAN configuration for
snapshot controlfile is set to default value which is
ORACLE_HOME/dbs/snapcf_.f. This is because
of the change in 11.2.0.2 that any of the nodes in the cluster can write
to the snapshot controlfile. Therefore, all nodes need access to
snapshot file which comes to solution that in a RAC environment snapshot
controlfile must be kept in ASM (I assume you use ASM, otherwise any
shared NFS or such).
So, what you need to do is:
1. First of all, check if it is really the case by :
RMAN> SHOW ALL ;
2. If it is, configure RMAN to use a shared resource by:
RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+RECOVERY/DB1/snapcf_db1.f';
"SYS@ORCL1> show parameter recovery
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest string +DGFRA
db_recovery_file_dest_size big integer 10000M
recovery_parallelism integer 0
MAN> show all;
RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET PARALLELISM 2;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK CONNECT '*' FORMAT '+DGRCAT/rmanbak/ORCL2_%U';
CONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT '*';
CONFIGURE CHANNEL 2 DEVICE TYPE DISK CONNECT '*';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/11.2.0/db_2/dbs/snapcf_ORCL1.f'; # default
RMAN> show snapshot controlfile name;
RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/11.2.0/db_2/dbs/snapcf_ORCL1.f'; # default
RMAN> configure snapshot controlfile name to '+DGFRA/ORCL/snapcf_ORCL1.f';
new RMAN configuration parameters:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DGFRA/ORCL/snapcf_ORCL1.f';
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete
RMAN> show snapshot controlfile name;
RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DGFRA/ORCL/snapcf_ORCL1.f';
RMAN> backup database plus archivelog;
Starting backup at 08-JAN-13
current log archived
using channel ORA_DISK_2
using channel ORA_DISK_1
channel ORA_DISK_1: starting compressed archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=2 sequence=55 RECID=87 STAMP=803847615
input archived log thread=1 sequence=43 RECID=88 STAMP=803858501
...
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
channel ORA_DISK_2: finished piece 1 at 08-JAN-13
piece handle=+DGFRA/orcl/backupset/2013_01_08/annnf0_tag20130108t151631_0.444.804179793 tag=TAG20130108T151631 comm ent=NONE
channel ORA_DISK_2: backup set complete, elapsed time: 00:00:01
Finished backup at 08-JAN-13
[grid@orarac1poc ~]$ asmcmd
ASMCMD> ls
ARCHIVELOG/
BACKUPSET/
CONTROLFILE/
FLASHBACK/
snapcf_orcl1.f
ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y BACKUPSET/
Y CONTROLFILE/
Y FLASHBACK/
N snapcf_orcl1.f => +DGFRA/ORCL/CONTROLFILE/Backup.463.804179619
--space before delete obsolete backup
SQL> select name, total_mb ,free_mb from v$asm_diskgroup;
NAME TOTAL_MB FREE_MB
------------------------------ ---------- ----------
DGDATA 10239 7587
DGFRA 10239 2487
OCR 2046 1650
DGRCAT 0 0
RMAN> delete noprompt obsolete;
RMAN retention policy will be applied to the command
RMAN retention policy is set to redundancy 1
using channel ORA_DISK_2
using channel ORA_DISK_1
Deleting the following obsolete backups and copies:
Type Key Completion Time Filename/Handle
-------------------- ------ ------------------ --------------------
Archive Log 1516 04-JAN-13 +DGFRA/orcl/archivelog/2013_01_04/thread_2_seq_55.499.803847615
Archive Log 1517 04-JAN-13 +DGFRA/orcl/archivelog/2013_01_04/thread_1_seq_43.501.803858501
Archive Log 1518 04-JAN-13 +DGFRA/orcl/archivelog/2013_01_04
...
deleted backup piece
backup piece handle=+DGFRA/orcl/backupset/2013_01_08/nnsnf0_tag20130108t150759_0.464.804179337 RECID=38 STAMP=804179337
Deleted 21 objects
SQL> select name, total_mb ,free_mb from v$asm_diskgroup;
NAME TOTAL_MB FREE_MB
------------------------------ ---------- ----------
DGDATA 10239 7587
DGFRA 10239 4035
OCR 2046 1650
DGRCAT 0 0
RMAN> list backup ;
...
