Tuesday, June 14, 2016

RMAN-06214: Archivelog , controlfile copy

1. for archived log file

RMAN-06207: WARNING: 677 objects could not be deleted for DISK channel(s) due
RMAN-06208:          to mismatched status.  Use CROSSCHECK command to fix status
RMAN-06210: List of Mismatched objects
RMAN-06211: ==========================
RMAN-06212:   Object Type   Filename/Handle
RMAN-06213: --------------- ---------------------------------------------------
RMAN-06214: Archivelog      /archive/testdb01/arch_822080339_219_1.arc
RMAN-06214: Archivelog      /archive/testdb01/arch_822080339_220_1.arc
RMAN-06214: Archivelog      /archive/testdb01/arch_822080339_221_1.arc
RMAN-06214: Archivelog      /archive/testdb01/arch_822080339_222_1.arc
RMAN-06214: Archivelog      /archive/testdb01/arch_822080339_223_1.arc
RMAN-06214: Archivelog      /archive/testdb01/arch_822080339_224_1.arc



This is simple to clear with one command: crosscheck archivelog all;

After which, just issue :  DELETE NOPROMPT OBSOLETE device type disk;



2. for controlfile copy, I have to crosscheck for each file copy name. i.e.

run {
allocate channel c1 ;
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_DAL_testdb01.2015516.210';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_AL_testdb01.2015517.30';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_DAL_testdb01.2015517.90';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_AL_testdb01.2015517.150';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_DAL_testdb01.2015517.210';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_AL_testdb01.2015518.30';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_DAL_testdb01.2015518.90';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_AL_testdb01.2015518.150';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_DAL_testdb01.2015518.210';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_AL_testdb01.2015519.30';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_DAL_testdb01.2015519.90';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_AL_testdb01.2015519.151';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_DAL_testdb01.2015519.210';
crosscheck controlfilecopy '/oracle/node1/admin/testdb01/cntrl/ctrlf.BILLDB21_AL_testdb01.2015520.30';

DELETE NOPROMPT OBSOLETE device type disk;

release channel c1;
}






Thursday, June 02, 2016

Ah, 11g OEM can't support my 12c PDB acting as Recovery Catalog


See the SID being used ?  In 12c, I don't see there is SID for 12c portable database. That is the problem.

NMO not setuid-root (Unix-only)

come across with above error when using Enterprise Manager, dealing with backup & recovery.

Referred to below two pages, to confirm they are same problem & solution.

https://community.oracle.com/thread/383358?start=0&tstart=0
https://dba010.wordpress.com/2011/07/18/error-connection-to-host-as-user-oracle-failed-error-nmo-not-setuid-root-unix-only/



My problem solved after ask system administrator to help run the root.sh too.


Monday, May 30, 2016

ORA-17628: Oracle error 17629 returned by remote Oracle server

when resync catalog

rman target / catalog rman/@RCAT

RMAN> RESYNC CATALOG FROM DB_UNIQUE_NAME DGs;


RMAN>
user interrupt received
RESYNC CATALOG FROM DB_UNIQUE_NAME DG;

resyncing from database with DB_UNIQUE_NAME DG
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync from db_unique_name command on default channel at 05/26/2016 16:19:54
ORA-17629: Cannot connect to the remote database server


changed to use password connection

orarac2poc:DGS:/home/oracle/ops/logs> rman target sys/ catalog rman/@RCAT

Recovery Manager: Release 11.2.0.4.0 - Production on Thu May 26 16:20:01 2016

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: DG (DBID=1716975309)
connected to recovery catalog database

RMAN> RESYNC CATALOG FROM DB_UNIQUE_NAME DG;

resyncing from database with DB_UNIQUE_NAME DG
starting full resync of recovery catalog
full resync complete

RMAN> RESYNC CATALOG FROM DB_UNIQUE_NAME all;


resyncing from database with DB_UNIQUE_NAME DG
starting full resync o

Friday, May 27, 2016

data guard SwitchOver using cloud control

When encounter this message in cloud control "You must reconnect to the current standby apply instance to perform the operation".

This means I have to connect to standby database as target, before the switch over can be performed. As currently I am connecting to primary database as target, which is not correct. 

wrong agent token when start cloud control agent

[symptoms]

orarac2poc:ORCL2:/home/oracle> . ./oms_agent_env
orarac2poc:ORCL2:/home/oracle> $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent .................................. failed.
Target Manager failed at Startup: targets.xml was rejected: loaded with a wrong agent token
Consult emctl.log and emagent.nohup in: /u01/app/oracle/agent12c/agent_inst/sysman/log


error message in emctl.log

UID        PID  PPID  C STIME TTY          TIME CMD
oracle   32708 29131  0 11:28 pts/3    00:00:00 /bin/sh -f /u01/app/oracle/agent12c/core/12.1.0.2.0/bin/emctl start agent

----
32717 :: Fri May 27 11:28:34 2016::EM_SECURE_HOSTNME: orarac2poc
32717 :: Fri May 27 11:28:34 2016::EM_SECURE_PORT: 3872
32717 :: Fri May 27 11:28:34 2016::EM_LISTEN_ON_ALL_NICS: true
32717 :: Fri May 27 11:28:36 2016::AgentLifeCycle.pm:status agent returned with retCode=1
32758 :: Fri May 27 11:28:36 2016::AgentLifeCycle.pm: Launching the watchdog process.
32758 :: Fri May 27 11:28:37 2016::TZ = Asia/Singapore, Read = Asia/Singapore. Found = 1.
32758 :: Fri May 27 11:28:37 2016::agentTZRegion successfully validated.
32758 :: Fri May 27 11:28:37 2016::AgentTuning.pm: Launching emdctl with -Xmx1024m
32758 :: Fri May 27 11:28:38 2016::AgentStatus.pm:emdctl run_autotune returned 0
32717 :: Fri May 27 11:28:36 2016::AgentLifeCycle.pm: Launched the watchdog process pid=32758
32717 :: Fri May 27 11:28:36 2016::AgentLifeCycle.pm: StartCEMD start retryCount=120
32717 :: Fri May 27 11:28:53 2016::AgentLifeCycle.pm: StartCEMD status message file detected
32717 :: Fri May 27 11:28:53 2016::Found Msg file with: {1464319733039,initializing}
32717 :: Fri May 27 11:28:53 2016::AgentLifeCycle.pm: StartCEMD Querying for the real status of the agent
32717 :: Fri May 27 11:28:57 2016::AgentLifeCycle.pm: StartCEMD status message file detected
32717 :: Fri May 27 11:28:57 2016::Found Msg file with: {1464319733039,initializing}
32717 :: Fri May 27 11:28:57 2016::AgentLifeCycle.pm: StartCEMD Querying for the real status of the agent
32717 :: Fri May 27 11:29:00 2016::AgentLifeCycle.pm:Watch dog processs id: 32758 exited with an exit code of 55
32717 :: Fri May 27 11:29:00 2016::AgentLifeCycle.pm: Exited loop retryCount=103 with retCode=1
32717 :: Fri May 27 11:29:00 2016::AgentLifeCycle.pm: StartCEMD Querying for the real status of the agent
32717 :: Fri May 27 11:29:02 2016::AgentLifeCycle.pm: Check agent status retCode=1
32717 :: Fri May 27 11:29:02 2016::TZ: EmctlLogAvailabilityMarker Operation=start Diag=failed


error message in emagent.nohup

 -----
 --- EMState agent
----- Fri May 27 11:28:37 2016::32758::Auto tuning the agent at time Fri May 27 11:28:37 2016 -----
----- Fri May 27 11:28:38 2016::32758::Finished auto tuning the agent at time Fri May 27 11:28:38 2016 -----
----- Fri May 27 11:28:38 2016::32758::Launching the JVM with following options: -Xmx140M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -----
----- Fri May 27 11:28:38 2016::337::Time elapsed between Launch of Watchdog process and execing EMAgent is 2 secs -----
----- Fri May 27 11:28:38 2016::32758::Agent Launched with PID 337 at time Fri May 27 11:28:38 2016 -----
2016-05-27 11:28:39,131 [1:main] WARN - Missing filename for log handler 'wsm'
2016-05-27 11:28:39,141 [1:main] WARN - Missing filename for log handler 'opss'
2016-05-27 11:28:39,142 [1:main] WARN - Missing filename for log handler 'opsscfg'
----- Fri May 27 11:28:58 2016::32758::Checking status of EMAgent : 337 -----
----- Fri May 27 11:28:58 2016::32758::EMAgent exited at Fri May 27 11:28:58 2016 with return value 55. -----
----- Fri May 27 11:28:58 2016::32758::EMAgent has exited due to initialization failure. -----
----- Fri May 27 11:28:58 2016::32758::Stopping other components. -----
----- Fri May 27 11:28:58 2016::32758::Commiting Process death. -----
----- Fri May 27 11:28:58 2016::32758::Generating additional diagnostics for 337 -----
----- Fri May 27 11:28:58 2016::32758::Generating openfiles report in /u01/app/oracle/agent12c/agent_inst/sysman/log/lsof_337 -----
----- Fri May 27 11:28:58 2016::32758::Generating netstat report in /u01/app/oracle/agent12c/agent_inst/sysman/log/netstat_337 -----
----- Fri May 27 11:28:59 2016::32758::Generating all host processes report in /u01/app/oracle/agent12c/agent_inst/sysman/log/allprocesses_337 -----
----- Fri May 27 11:28:59 2016::32758::Exiting watchdog loop
targets.xml was rejected: loaded with a wrong agent token


when configure recovery catalog in cloud control

Validation Error
Examine and correct the following errors, then retry the operation:
Host Credentials - unable to connect to the agent at https://orarac2poc:3872/emd/main/ [Connection refused]


[solution]

found the advice in https://community.oracle.com/thread/3674803?start=0&tstart=0

--> Move your existing target.xml file
mv target.xml target.xml.old


--> Create an empty targets.xml file under /AGENT_INST/sysman/emd with the below content


--> Start the agent


$AGENT_INST/bin/emctl start agent
This starts agent but without any targets populated in targets.xml


To discover targets perform agent resynchronization operation from console



orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> cat targets.xml

       
               
               
               
               
               
               
               
               
               
               
...

       
       
               
               
               
               
               
               
               
       

orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> mv  targets.xml targets.xml.bak
orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> vi targets.xml
orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> cat targets.xml






orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent .............. started.


[VERIFICATION]

orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> netstat -an |grep 3872 |grep LIST
tcp        0      0 :::3872                     :::*                        LISTEN
unix  2      [ ACC ]     STREAM     LISTENING     397387209 /var/tmp/.oracle/sOracle_CSS_LclLstnr_clus_2
unix  2      [ ACC ]     STREAM     LISTENING     397387213 /var/tmp/.oracle/sOCSSD_LL_orarac2poc_clus



orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> cat targets.xml



AND after I add one database as managed target in cloud control, I notice the content of targets.xml got changed.


after a while ,prompt RCAT


orarac2poc:ORCL2:/u01/app/oracle/agent12c/agent_inst/sysman/emd> cat targets.xml

       
               
               
               
               
               
               
               
       









Wednesday, May 25, 2016

RMAN ORA-19606: Cannot copy or restore to snapshot control file

When RMAN needs to resynchronize the recovery catalog with a read-consistent version of the control file, it creates a temporary snapshot control file. RMAN needs a snapshot control file when resynchronizing with the recovery catalog or when making a backup of the current control file.
The default location for the snapshot control file is platform-specific and depends on the Oracle home of each target database. For example, the default filename on some Linux platforms is $ORACLE_HOME/dbs/snapcf_@.f


RMAN> delete obsolete;

starting full resync of recovery catalog
full resync complete
RMAN retention policy will be applied to the command
RMAN retention policy is set to redundancy 3
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=188 device type=DISK
Deleting the following obsolete backups and copies:
Type                 Key    Completion Time    Filename/Handle
-------------------- ------ ------------------ --------------------
Control File Copy     1123728 2016-JAN-19 14:33:21 /software/oratest/product/11.2.0/dbs/snapcf_TESTDB.f
Backup Set           1123715 2015-NOV-06 10:55:47

Do you really want to delete the above objects (enter YES or NO)? yes
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of delete command on ORA_DISK_1 channel at 05/25/2016 11:57:24
ORA-19606: Cannot copy or restore to snapshot control file


SOLUTION

Not sure why this happens, but the solution is to configure a different snapshot controlfile name so that RMAN can use that, thus allowing you to remove the “old” one. Then configure back to what you had:
configure snapshot controlfile name to '/tmp/it.f';
crosscheck controlfilecopy "/oracle/product/11.2.0.2/dbs/snapcf_P10AC.f";
delete expired controlfilecopy "/oracle/product/11.2.0.2/dbs/snapcf_P10AC.f";
delete noprompt obsolete;
configure snapshot controlfile name to '/oracle/product/11.2.0.2/dbs/snapcf_P10AC.f';