ORA-17628: Oracle error 17629 returned by remote Oracle server

when resync catalog

rman target / catalog rman/@RCAT

RMAN> RESYNC CATALOG FROM DB_UNIQUE_NAME DGs;


RMAN>
user interrupt received
RESYNC CATALOG FROM DB_UNIQUE_NAME DG;

resyncing from database with DB_UNIQUE_NAME DG
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync from db_unique_name command on default channel at 05/26/2016 16:19:54
ORA-17629: Cannot connect to the remote database server


changed to use password connection

orarac2poc:DGS:/home/oracle/ops/logs> rman target sys/ catalog rman/@RCAT

Recovery Manager: Release 11.2.0.4.0 - Production on Thu May 26 16:20:01 2016

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: DG (DBID=1716975309)
connected to recovery catalog database

RMAN> RESYNC CATALOG FROM DB_UNIQUE_NAME DG;

resyncing from database with DB_UNIQUE_NAME DG
starting full resync of recovery catalog
full resync complete

RMAN> RESYNC CATALOG FROM DB_UNIQUE_NAME all;


resyncing from database with DB_UNIQUE_NAME DG
starting full resync o

data guard SwitchOver using cloud control

When encounter this message in cloud control "You must reconnect to the current standby apply instance to perform the operation".

This means I have to connect to standby database as target, before the switch over can be performed. As currently I am connecting to primary database as target, which is not correct. 

wrong agent token when start cloud control agent

[symptoms]

orarac2poc:ORCL2:/home/oracle> . ./oms_agent_env
orarac2poc:ORCL2:/home/oracle> $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent .................................. failed.
Target Manager failed at Startup: targets.xml was rejected: loaded with a wrong agent token
Consult emctl.log and emagent.nohup in: /u01/app/oracle/agent12c/agent_inst/sysman/log


error message in emctl.log

UID        PID  PPID  C STIME TTY          TIME CMD
oracle   32708 29131  0 11:28 pts/3    00:00:00 /bin/sh -f /u01/app/oracle/agent12c/core/12.1.0.2.0/bin/emctl start agent

----
32717 :: Fri May 27 11:28:34 2016::EM_SECURE_HOSTNME: orarac2poc
32717 :: Fri May 27 11:28:34 2016::EM_SECURE_PORT: 3872
32717 :: Fri May 27 11:28:34 2016::EM_LISTEN_ON_ALL_NICS: true
32717 :: Fri May 27 11:28:36 2016::AgentLifeCycle.pm:status agent returned with retCode=1
32758 :: Fri May 27 11:28:36 2016::AgentLifeCycle.pm: Launching the watchdog process.
32758 :: Fri May 27 11:28:37 2016::TZ = Asia/Singapore, Read = Asia/Singapore. Found = 1.
32758 :: Fri May 27 11:28:37 2016::agentTZRegion successfully validated.
32758 :: Fri May 27 11:28:37 2016::AgentTuning.pm: Launching emdctl with -Xmx1024m
32758 :: Fri May 27 11:28:38 2016::AgentStatus.pm:emdctl run_autotune returned 0
32717 :: Fri May 27 11:28:36 2016::AgentLifeCycle.pm: Launched the watchdog process pid=32758
32717 :: Fri May 27 11:28:36 2016::AgentLifeCycle.pm: StartCEMD start retryCount=120
32717 :: Fri May 27 11:28:53 2016::AgentLifeCycle.pm: StartCEMD status message file detected
32717 :: Fri May 27 11:28:53 2016::Found Msg file with: {1464319733039,initializing}
32717 :: Fri May 27 11:28:53 2016::AgentLifeCycle.pm: StartCEMD Querying for the real status of the agent
32717 :: Fri May 27 11:28:57 2016::AgentLifeCycle.pm: StartCEMD status message file detected
32717 :: Fri May 27 11:28:57 2016::Found Msg file with: {1464319733039,initializing}
32717 :: Fri May 27 11:28:57 2016::AgentLifeCycle.pm: StartCEMD Querying for the real status of the agent
32717 :: Fri May 27 11:29:00 2016::AgentLifeCycle.pm:Watch dog processs id: 32758 exited with an exit code of 55
32717 :: Fri May 27 11:29:00 2016::AgentLifeCycle.pm: Exited loop retryCount=103 with retCode=1
32717 :: Fri May 27 11:29:00 2016::AgentLifeCycle.pm: StartCEMD Querying for the real status of the agent
32717 :: Fri May 27 11:29:02 2016::AgentLifeCycle.pm: Check agent status retCode=1
32717 :: Fri May 27 11:29:02 2016::TZ: EmctlLogAvailabilityMarker Operation=start Diag=failed


error message in emagent.nohup

 -----
 --- EMState agent
----- Fri May 27 11:28:37 2016::32758::Auto tuning the agent at time Fri May 27 11:28:37 2016 -----
----- Fri May 27 11:28:38 2016::32758::Finished auto tuning the agent at time Fri May 27 11:28:38 2016 -----
----- Fri May 27 11:28:38 2016::32758::Launching the JVM with following options: -Xmx140M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -----
----- Fri May 27 11:28:38 2016::337::Time elapsed between Launch of Watchdog process and execing EMAgent is 2 secs -----
----- Fri May 27 11:28:38 2016::32758::Agent Launched with PID 337 at time Fri May 27 11:28:38 2016 -----
2016-05-27 11:28:39,131 [1:main] WARN - Missing filename for log handler 'wsm'
2016-05-27 11:28:39,141 [1:main] WARN - Missing filename for log handler 'opss'
2016-05-27 11:28:39,142 [1:main] WARN - Missing filename for log handler 'opsscfg'
----- Fri May 27 11:28:58 2016::32758::Checking status of EMAgent : 337 -----
----- Fri May 27 11:28:58 2016::32758::EMAgent exited at Fri May 27 11:28:58 2016 with return value 55. -----
----- Fri May 27 11:28:58 2016::32758::EMAgent has exited due to initialization failure. -----
----- Fri May 27 11:28:58 2016::32758::Stopping other components. -----
----- Fri May 27 11:28:58 2016::32758::Commiting Process death. -----
----- Fri May 27 11:28:58 2016::32758::Generating additional diagnostics for 337 -----
----- Fri May 27 11:28:58 2016::32758::Generating openfiles report in /u01/app/oracle/agent12c/agent_inst/sysman/log/lsof_337 -----
----- Fri May 27 11:28:58 2016::32758::Generating netstat report in /u01/app/oracle/agent12c/agent_inst/sysman/log/netstat_337 -----
----- Fri May 27 11:28:59 2016::32758::Generating all host processes report in /u01/app/oracle/agent12c/agent_inst/sysman/log/allprocesses_337 -----
----- Fri May 27 11:28:59 2016::32758::Exiting watchdog loop
targets.xml was rejected: loaded with a wrong agent token


when configure recovery catalog in cloud control

Validation Error
Examine and correct the following errors, then retry the operation:
Host Credentials - unable to connect to the agent at https://orarac2poc:3872/emd/main/ [Connection refused]


[solution]

found the advice in https://community.oracle.com/thread/3674803?start=0&tstart=0

--> Move your existing target.xml file

mv target.xml target.xml.old

--> Create an empty targets.xml file under /AGENT_INST/sysman/emd with the below content

--> Start the agent

$AGENT_INST/bin/emctl start agent

This starts agent but without any targets populated in targets.xml

To discover targets perform agent resynchronization operation from console

orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> cat targets.xml

       
               
               
               
               
               
               
               
               
               
               
...

       
       
               
               
               
               
               
               
               
       

orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> mv  targets.xml targets.xml.bak
orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> vi targets.xml
orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> cat targets.xml






orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent .............. started.


[VERIFICATION]

orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> netstat -an |grep 3872 |grep LIST
tcp        0      0 :::3872                     :::*                        LISTEN
unix  2      [ ACC ]     STREAM     LISTENING     397387209 /var/tmp/.oracle/sOracle_CSS_LclLstnr_clus_2
unix  2      [ ACC ]     STREAM     LISTENING     397387213 /var/tmp/.oracle/sOCSSD_LL_orarac2poc_clus



orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> cat targets.xml



AND after I add one database as managed target in cloud control, I notice the content of targets.xml got changed.


after a while ,prompt RCAT


orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> cat targets.xml

       
               
               
               
               
               
               
               
       

RMAN ORA-19606: Cannot copy or restore to snapshot control file

When RMAN needs to resynchronize the recovery catalog with a read-consistent version of the control file, it creates a temporary snapshot control file. RMAN needs a snapshot control file when resynchronizing with the recovery catalog or when making a backup of the current control file.

The default location for the snapshot control file is platform-specific and depends on the Oracle home of each target database. For example, the default filename on some Linux platforms is $ORACLE_HOME/dbs/snapcf_@.f. 

RMAN> delete obsolete;

starting full resync of recovery catalog

full resync complete

RMAN retention policy will be applied to the command

RMAN retention policy is set to redundancy 3

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=188 device type=DISK

Deleting the following obsolete backups and copies:

Type                 Key    Completion Time    Filename/Handle

-------------------- ------ ------------------ --------------------

Control File Copy     1123728 2016-JAN-19 14:33:21 /software/oratest/product/11.2.0/dbs/snapcf_TESTDB.f

Backup Set           1123715 2015-NOV-06 10:55:47

Do you really want to delete the above objects (enter YES or NO)? yes

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of delete command on ORA_DISK_1 channel at 05/25/2016 11:57:24

ORA-19606: Cannot copy or restore to snapshot control file

SOLUTION

Not sure why this happens, but the solution is to configure a different snapshot controlfile name so that RMAN can use that, thus allowing you to remove the “old” one. Then configure back to what you had:

configure snapshot controlfile name to '/tmp/it.f';
crosscheck controlfilecopy "/oracle/product/11.2.0.2/dbs/snapcf_P10AC.f";
delete expired controlfilecopy "/oracle/product/11.2.0.2/dbs/snapcf_P10AC.f";
delete noprompt obsolete;
configure snapshot controlfile name to '/oracle/product/11.2.0.2/dbs/snapcf_P10AC.f';

ORA-19806: cannot make duplex backups in recovery area

when running "backup filesperset 10 archivelog all not backed up 2 times;", encountered below error :

channel ORA_DISK_1: starting piece 1 at 2016-MAY-25 14:59:18
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/25/2016 14:59:18
ORA-19806: cannot make duplex backups in recovery area

Initially thought FRA is full, but it is not.

SYS@PELD> @@chk_fra_usage

NAME

------------------------------------------------------------------------------------------------------------------------------------

SPACE_LIMIT SPACE_USED SPACE_RECLAIMABLE NUMBER_OF_FILES

----------- ---------- ----------------- ---------------

/dbfra/testdb01d

 5.3687E+10  959854592                 0              20

FILE_TYPE            PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES

-------------------- ------------------ ------------------------- ---------------

CONTROL FILE                          0                         0               0

REDO LOG                              0                         0               0

ARCHIVED LOG                          0                         0               0

BACKUP PIECE                       1.79                         0              20

IMAGE COPY                            0                         0               0

FLASHBACK LOG                         0                         0               0

FOREIGN ARCHIVED LOG                  0                         0               0

7 rows selected.

Googled and found below in http://www.allinterview.com/showanswers/41216/ora-19806-cannot-make-duplex-backups-in-recovery-area.html

this is because you might have set the parametes
1)configure datafile backup copies for device type disk to
2)configure archivelog backup copies for device type disk to

now the solution is to clear the above two configuration
settings i,e...
A)
1)configure datafile backup copies for device type disk to
clear;
2)configure archivelog backup copies for device type disk to
clear;
now it will work ................... 

Tried to reset back to 1 copy, and it did works for me now. 

RMAN> CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK clear;

old RMAN configuration parameters:

CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 2;

RMAN configuration parameters are successfully reset to default value

RMAN can duplex backups to either disk or tape, but cannot duplex backups to tape and disk simultaneously. When backing up to tape, ensure that the number of copies does not exceed the number of available tape devices. 

This is could be related to I am using FRA as backup destination.

simpler error message for RMAN tested against NOARCHIVELOG in 12c

Before actual cold backup for development database, I skip the shutdown command to test the script deployment, without actually bring down the database (during office hour).

I expected to see below error message prior to 12c.

RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/04/2016 17:28:14
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode
continuing other job steps, job failed will not be re-run

However, this is repeated for each backup set while travelling all data files.

channel ORA_DISK_1: SID=244 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=251 device type=DISK
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/04/2016 17:28:14
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/04/2016 17:28:14
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/04/2016 17:28:14
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/04/2016 17:28:14
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/04/2016 17:28:14
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/04/2016 17:28:15
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/04/2016 17:28:15
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode
continuing other job steps, job failed will not be re-run
channel ORA_DISK_2: starting compressed full datafile backup set
channel ORA_DISK_2: specifying datafile(s) in backup set
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_2 channel at 05/04/2016 17:28:15
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode

Recovery Manager complete.

Good to see , it is getting simpler in 12c as shown below, just one word.

Starting backup at 2016-MAY-20 17:53:10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=368 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=250 device type=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: SID=366 device type=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: SID=131 device type=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 05/20/2016 17:53:13
RMAN-06149: cannot BACKUP DATABASE in NOARCHIVELOG mode

Recovery Manager complete.

reduce AWR retention but hit ORA-13541

Need to reduce AWR retention due to no disk space in this development database.

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> select * from dba_hist_wr_control;

      DBID
----------
SNAP_INTERVAL
---------------------------------------------------------------------------
RETENTION
---------------------------------------------------------------------------
TOPNSQL        CON_ID
---------- ----------
2723810418
+00000 01:00:00.0
+00042 00:00:00.0
DEFAULT             0




SQL> begin DBMS_WORKLOAD_REPOSITORY.MODIFY_SNAPSHOT_SETTINGS(11520,60); end;
  2
  3  /
begin DBMS_WORKLOAD_REPOSITORY.MODIFY_SNAPSHOT_SETTINGS(11520,60); end;
*
ERROR at line 1:
ORA-13541: system moving window baseline size (3628800) greater than retention
(691200)
ORA-06512: at "SYS.DBMS_WORKLOAD_REPOSITORY", line 198
ORA-06512: at "SYS.DBMS_WORKLOAD_REPOSITORY", line 246
ORA-06512: at line 1


SQL> SELECT moving_window_size
FROM   dba_hist_baseline
WHERE  baseline_type = 'MOVING_WINDOW';


MOVING_WINDOW_SIZE
------------------
                42

SQL>
SQL> exec DBMS_WORKLOAD_REPOSITORY.modify_baseline_window_size( window_size =>7);

PL/SQL procedure successfully completed.

SQL> SELECT moving_window_size
FROM   dba_hist_baseline
  2    3  WHERE  baseline_type = 'MOVING_WINDOW';

MOVING_WINDOW_SIZE
------------------
                 7

SQL>
SQL> begin DBMS_WORKLOAD_REPOSITORY.MODIFY_SNAPSHOT_SETTINGS(11520,60); end;
  2  /

PL/SQL procedure successfully completed.

SQL>  select * from dba_hist_wr_control;

      DBID
----------
SNAP_INTERVAL
---------------------------------------------------------------------------
RETENTION
---------------------------------------------------------------------------
TOPNSQL        CON_ID
---------- ----------
2723810418
+00000 01:00:00.0
+00008 00:00:00.0
DEFAULT             0


References:
http://www.sqlpanda.com/2013/02/ora-13541-system-moving-window-baseline.html

rman shell script but not connecting to recovery catalog

I have one RMAN backup script works fine , but when running for a 12c.1 database, it keeps showing me below in logfile

"using target database control file instead of recovery catalog
Report of database schema for database with db_unique_name DEV"


And this the working fine shell function.


func_cold_db_backup() {
${ORACLE_HOME}/bin/rman target /  cmdfile='/oracle/UAT/ops/exec/cold_backup.rman' log=$logfile <>$logfile 2>&1
${ORACLE_HOME}/bin/rman target /   log=$logfile <>$logfile 2>&1
connect catalog rman/pwdxxxxx@RCAT
@/oracle/UAT/ops/exec/cold_backup.rman
EOF


When manually try the command , it works fine and shows me connected to RCAT.

${ORACLE_HOME}/bin/rman target /  catalog rman@RCAT

connected to target database: UAT (DBID=27281018)

connected to recovery catalog database

So the only difference is cmdfile parameter.

As such, I rewrite the function and works "better" now for this database.

func_cold_db_backup() {
${ORACLE_HOME}/bin/rman target /   log=$logfile <>$logfile 2>&1
connect catalog rman/pwdxxx@RCAT
@/oracle/UAT/ops/exec/cold_backup.rman
EOF

two ways to decommission a database backed up by RMAN

1. If follow proper procedure, backups should be deleted before unregistrating from recovery catalog

Either using with RMAN command "delete backup", or
during dropping database with "drop database including backups"

2. However, if we forget to follow above sequence, we still have manual way to achieve the same.

2.1  Manually delete backups from DISK, usually they should be under FRA.
2.2 Connect to catalog database without specifying target

     rman rman/password@RCAT

    > set dbid=
    > unregister database;



e.g.




> rman  catalog rman@RCAT

Recovery Manager: Release 11.2.0.3.0 - Production on Fri May 13 15:20:03 2016

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

recovery catalog database Password:
connected to recovery catalog database

RMAN> set dbid=649569281;

executing command: SET DBID
database name is "CTCG" and DBID is 649569281

RMAN> unregister database;

database name is "CTCG" and DBID is 649569281

Do you really want to unregister the database (enter YES or NO)? yes
database unregistered from the recovery catalog

RMAN> exit

double-quoted not accept in 10g rman, but not 11g

In my cold backup shell script, there is below line, which works fine in 11g.

A double quote inside single quote , to pass ORACLE_HOME path  to &1 of cold_backup.rman

${ORACLE_HOME}/bin/rman target /  cmdfile='./cold_backup.rman "/software/oraabc/product/11.2.0"' log=$logfile <>$logfile 2>&1
connect catalog rman/xxxxxx@RCAT1
EOF

But it is giving me below error , when running for 10g database.

Both single and double quotes (' or ") are accepted for a quoted-string.

Quotes are not required unless the string contains embedded white-space.

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-00552: syntax error in command line arguments

RMAN-01009: syntax error: found "double-quoted-string": expecting one of: "append, at, auxiliary, catalog, cmdfile, clone, checksyntax, debug, log, msglog, mask, msgno, nocatalog, pipe, rcvcat, script, slaxdebug, send, target, timeout, trace"

RMAN-01007: at line 0 column 52 file: command line arguments

Recovery Manager: Release 10.2.0.2.0 - Production on Wed May 11 15:57:27 2016

This is a funny finding.

NFS error in AIX when running RMAN backup to disk

channel ORA_DISK_1 disabled, job failed on it will be run on another channel
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_2 channel at 05/04/2016 21:01:22
ORA-19504: failed to create file "/dbfra/testdb11d/bbbb/BBBB/backupset/2016_05_04/o1_mf_nnndf_TAG20160504T210121_%u_.bkp"
ORA-27054: NFS file system where the file is created or resides is not mounted with correct options
Additional information: 5
Additional information: 9


although the file system is presented in /etc/filesystems file.

/dbfra:
        dev             = "/dbfra"
        vfs             = nfs
        nodename        = 10.149.90.205
        mount           = true
        options         = bg,soft,intr,rsize=32768,wsize=32768,proto=tcp,sec=sys
        account         = false



resolved by add below parameter to pfile.

echo 'event="10298 trace name context forever, level 32"'  >>initBBBB.ora