Wednesday, September 04, 2013

answer for minimal downtime to replatform a large database

conclusions of testing with incremental updated image copy.

1. if using current control file for recovery, the CURRENT online log are also needed. then open without RESETLOGS


2. If using backup controlfile , then needs to recover with "recover database noredo", followed by open RESETLOGS.

3. If using backup controlfile, once restored from datafilecopy, the COPY's info is no longer insided controlfile.  needs to re-create level0 datafilecopy after recovery, open RESETLOGS.

4. to minimize the downtime for large (TB level) database, can pre-copy a baseline datafiles via NFS for example, followed by rolling forward with incremental archived log files and current control files plus CURRENT online log files.

12c cloud control log file housekeep.

orarac2poc:ORCL2:/home/oracle> df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      16220072   8313944   7068908  55% /
/dev/sda1               101086     21079     74788  22% /boot
tmpfs                  2097152   1169780    927372  56% /dev/shm
/dev/sdb1             36116556  33641364    640572  99% /u01

Googled, followed this article , and did minor enhancement for cron.

#!/bin/sh

export OMS_HOME=/u01/app/oracle/oms12c
export AGENT_HOME=/u01/app/oracle/agent12c

#find and clean Oracle 12c Agent log files older then 14 days
find ${AGENT_HOME}/agent_inst/sysman/log/ -name "gcagent.log.*" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null

#find and clean Oracle 12c OMS log files older then 14 days
find ${OMS_HOME}/gc_inst/em/EMGC_OMS1/sysman/log/ -name "emoms.trc*" -mtime +14 -print|xargs rm  >/dev/null 2>/dev/null
find ${OMS_HOME}/gc_inst/em/EMGC_OMS1/sysman/log/ -name "emctl.log.*" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null
find ${OMS_HOME}/gc_inst/em/EMGC_OMS1/sysman/log/ -name "oms_diag_info*.msg" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null



#find and clean Oracle 12c Oracle HTTP Server (OHS) log files older then 14 days
find ${OMS_HOME}/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/  -name "access_log*" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null
find ${OMS_HOME}/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/  -name "em_upload_https_access_log*" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null
find ${OMS_HOME}/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/  -name "ohs1-*.log" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null

#find and clean Oracle 12c Oracle WebLogic log files older then 14 days
find ${OMS_HOME}/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/ -name "access.log*" -mtime +14 -print|xargs rm  >/dev/null 2>/dev/null
find ${OMS_HOME}/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/ -name "EMGC_OMS1.log*" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null
find ${OMS_HOME}/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/ -name "EMGC_OMS1.out*" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null
find ${OMS_HOME}/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/ -name "EMGC_OMS1-diagnostic*.log" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null
find ${OMS_HOME}/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs -name "EMGC_ADMINSERVER.out*" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null
find ${OMS_HOME}/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs -name "GCDomain.log*" -mtime +14 -print|xargs rm >/dev/null 2>/dev/null

#find and clean Oracle 12c OPMN log files older then 14 days
#${OMS_HOME}/gc_inst/WebTierIH1/diagnostics/logs/OPMN/opmn

#find and clean Oracle 12c diagnose folder in oracle home log files older then 14 days
#find /home/oracle/oradiag_oracle/diag/clients/user_oracle/host_1387324873_11/alert  -name "log_*.xml" -mtime +14 -print|xargs rm

#the next line is commented for know because on 12c I disabled the module according to metalink note:1396472.1
#cat /u01/oracle/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/mod_wl_ohs.log>/u01/oracle/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/mod_wl_ohs_$DATE_VAR.log >/u01/oracle/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/mod_wl_ohs.log


orarac2poc:ORCL2:/u01/app> df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      16220072   8314016   7068836  55% /
/dev/sda1               101086     21079     74788  22% /boot
tmpfs                  2097152   1169780    927372  56% /dev/shm
/dev/sdb1             36116556  31533108   2748828  92% /u01

I also followed below doc to refine the setting for this poc server.
12c Cloud Control: How to Enable Log Rotation Policy to Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files? (Doc ID 1450535.1)

spent some time to find out the port number of weblogic administration console.  it is 7101 , not 7102 mentioned in above metalink doc.


orarac2poc:RCAT:/u01/app/oracle/oms12c> find ./ -name portlist.ini
./oms/install/portlist.ini
orarac2poc:RCAT:/u01/app/oracle/oms12c> cat ./oms/install/portlist.ini

orarac2poc:RCAT:/u01/app/oracle/oms12c> find ./ -name *.ini
./.gcinstall_temp/staticports.ini
./oracle_common/oui/oraparam.ini
./oracle_common/oui/clusterparam.ini
./oracle_common/OPatch/opatch.ini
./oracle_common/inventory/Templates/oui/oraparam.ini
./oracle_common/inventory/Templates/oui/clusterparam.ini
./oracle_common/inventory/Templates/OPatch/opatch.ini
./Oracle_WT/oui/oraparam.ini
./Oracle_WT/oui/clusterparam.ini
./Oracle_WT/OPatch/opatch.ini
./Oracle_WT/inventory/Templates/oui/oraparam.ini
./Oracle_WT/inventory/Templates/oui/clusterparam.ini
./Oracle_WT/inventory/Templates/OPatch/opatch.ini
./oms/install/portlist.ini
./oms/install/installer/oraparam.ini
./oms/oui/oraparam.ini
./oms/oui/clusterparam.ini
./oms/OPatch/opatch.ini
./oms/inventory/Scripts/ext/lib/portrange.ini
./oms/inventory/Scripts/ext/lib/portrange_upgrade.ini
./oms/inventory/backup/2013-01-15_04-39-32-PM/Scripts/ext/lib/portrange.ini
./oms/inventory/backup/2013-01-15_04-39-32-PM/Scripts/ext/lib/portrange_upgrade.ini
./oms/inventory/Templates/install/installer/oraparam.ini
./oms/inventory/Templates/oui/oraparam.ini
./oms/inventory/Templates/oui/clusterparam.ini
./oms/inventory/Templates/OPatch/opatch.ini
./jdk16/inventory/Scripts/ext/lib/portrange.ini
./jdk16/inventory/Scripts/ext/lib/portrange_upgrade.ini
./jdk16/inventory/backup/2013-01-15_04-28-26PM/Scripts/ext/lib/portrange.ini
./jdk16/inventory/backup/2013-01-15_04-28-26PM/Scripts/ext/lib/portrange_upgrade.ini



orarac2poc:RCAT:/u01/app/oracle/oms12c> cat ./.gcinstall_temp/staticports.ini
Enterprise Manager Upload Http Port=4889
Enterprise Manager Upload Http SSL Port=4900
Enterprise Manager Central Console Http SSL Port=7799
Node Manager Http SSL Port=7403
Managed Server Http Port=7202
Enterprise Manager Central Console Http Port=7788
Oracle Management Agent Port=3872
Admin Server Http SSL Port=7101
Managed Server Http SSL Port=7301

https://10.139.90.42:7101/console

Monday, September 02, 2013

differences of log_archive_dest and log_archive_dest_n

SYS@DG> alter system set log_archive_dest = '/home/oracle/temp';
alter system set log_archive_dest = '/home/oracle/temp'
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-16018: cannot use LOG_ARCHIVE_DEST with LOG_ARCHIVE_DEST_n or DB_RECOVERY_FILE_DEST


orarac1poc:DG:/u01/app/oracle/product/11.2.0/db_2/dbs> grep -i log_archive initDG.ora
*.LOG_ARCHIVE_CONFIG='DG_CONFIG=(DG,DGS)'
*.LOG_ARCHIVE_DEST_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=DG'
*.LOG_ARCHIVE_DEST_2='SERVICE=DGS ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DGS'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='DG_%t_%s_%r.arc'


SYS@DG> alter system set LOG_ARCHIVE_DEST_3='LOCATION=/home/oracle/temp/' ;

System altered.

SYS@DG> !ls -l /home/oracle/temp/
total 0

SYS@DG> alter system switch logfile;

System altered.

SYS@DG> !ls -l /home/oracle/temp/
total 26948
-rw-r----- 1 oracle asmadmin 27561984 Sep  2 09:57 DG_1_2731_805314423.arc

SYS@DG> !ls -l /u01/oradata/DG/FRA/DG/archivelog/2013_09_02/
total 107168
-rw-r----- 1 oracle asmadmin 41866752 Sep  2 00:02 o1_mf_1_2729_926s97rm_.arc
-rw-r----- 1 oracle asmadmin 40182784 Sep  2 04:00 o1_mf_1_2730_92777d27_.arc
-rw-r----- 1 oracle asmadmin 27561984 Sep  2 09:57 o1_mf_1_2731_927w3lm5_.arc


LOG_ARCHIVE_DEST is applicable only if you are running the database in ARCHIVELOG mode or are recovering a database from archived redo logs. LOG_ARCHIVE_DEST is incompatible with the LOG_ARCHIVE_DEST_n parameters, and must be defined as the null string ("") or (' ') when any LOG_ARCHIVE_DEST_n parameter has a value other than a null string.

LOG_ARCHIVE_FORMAT is applicable only if you are using the redo log in ARCHIVELOG mode. Use a text string and variables to specify the default filename format when archiving redo log files. The string generated from this format is appended to the string specified in the LOG_ARCHIVE_DEST parameter.

RMAN-03009 when CONTROLFILE AUTOBACKUP ON

RMAN> show all;

RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F_%U';
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET PARALLELISM 1;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT '*';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DGFRA/ORCL/snapcf_ORCL1.f';


Starting Control File and SPFILE Autobackup at Sep 02 2013 16:43:09
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of Control File and SPFILE Autobackup command on ORA_DISK_1 channel at 09/02/2013 16:43:12
ORA-19715: invalid format c for generated name
ORA-27302: failure occurred at: slgpn


but no problem with controlfile autobackup off.


RMAN> CONFIGURE CONTROLFILE AUTOBACKUP off;

old RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP OFF;
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete

RMAN> backup current controlfile;

Starting backup at Sep 02 2013 16:59:16
using channel ORA_DISK_1
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at Sep 02 2013 16:59:20
channel ORA_DISK_1: finished piece 1 at Sep 02 2013 16:59:21
piece handle=+DGFRA/orcl/backupset/2013_09_02/ncnnf0_tag20130902t165916_0.291.825094761 tag=TAG20130902T165916 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at Sep 02 2013 16:59:21


after did some research, realized something wrong with the format, i.e. the full path was not specified.

RMAN> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+DGFRA/ORCL/ctrl_%F';

old RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F_%U';
new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+DGFRA/ORCL/ctrl_%F';
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete

RMAN> CONFIGURE CONTROLFILE AUTOBACKUP on;

old RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP OFF;
new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete

RMAN> backup current controlfile;

Starting backup at Sep 02 2013 17:06:20
using channel ORA_DISK_1
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at Sep 02 2013 17:06:23
channel ORA_DISK_1: finished piece 1 at Sep 02 2013 17:06:24
piece handle=+DGFRA/orcl/backupset/2013_09_02/ncnnf0_tag20130902t170621_0.555.825095183 tag=TAG20130902T170621 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at Sep 02 2013 17:06:24

Starting Control File and SPFILE Autobackup at Sep 02 2013 17:06:24
piece handle=+DGFRA/orcl/ctrl_c-1330909046-20130902-03 comment=NONE
Finished Control File and SPFILE Autobackup at Sep 02 2013 17:06:27

RMAN> show all;

RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+DGFRA/ORCL/ctrl_%F';
CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET PARALLELISM 1;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK CONNECT '*' FORMAT   '+DGRCAT/rmanbak/ORCL2_%U';
CONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT '*';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DGFRA/ORCL/snapcf_ORCL1.f';

RMAN> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+DGFRA/ORCL/AUTOBACKUP/%F';

old RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+DGFRA/ORCL/ctrl_%F';
new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+DGFRA/ORCL/AUTOBACKUP/%F';
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete

RMAN> backup database;

Starting backup at Sep 02 2013 17:10:45
using channel ORA_DISK_1
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00002 name=+DGDATA/orcl/sysaux01.dbf
input datafile file number=00001 name=+DGDATA/orcl/system01.dbf
input datafile file number=00005 name=+DGDATA/orcl/example01.dbf
input datafile file number=00003 name=+DGDATA/orcl/undotbs01.dbf
input datafile file number=00006 name=+DGDATA/orcl/undotbs02.dbf
input datafile file number=00004 name=+DGDATA/orcl/users01.dbf
channel ORA_DISK_1: starting piece 1 at Sep 02 2013 17:10:47
channel ORA_DISK_1: finished piece 1 at Sep 02 2013 17:13:02
piece handle=+DGFRA/orcl/backupset/2013_09_02/nnndf0_tag20130902t171045_0.603.825095447 tag=TAG20130902T171045 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:02:15
Finished backup at Sep 02 2013 17:13:02

Starting Control File and SPFILE Autobackup at Sep 02 2013 17:13:02
piece handle=+DGFRA/orcl/autobackup/c-1330909046-20130902-05 comment=NONE
Finished Control File and SPFILE Autobackup at Sep 02 2013 17:13:06

RMAN> show all;

RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+DGFRA/ORCL/AUTOBACKUP/%F';
CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET PARALLELISM 1;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK CONNECT '*' FORMAT   '+DGRCAT/rmanbak/ORCL2_%U';
CONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT '*';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DGFRA/ORCL/snapcf_ORCL1.f';