Reconfigure 11gR2 clusterware services

Due to the following problem , which is likely a un-fixed bug, and I didn't use md_backup ASM info, thus  I have to re-create the ASM instance.

Fri Jan 25 22:00:40 2013
NOTE: attached to recovery domain 1
NOTE: starting recovery of thread=1 ckpt=32.568 group=1 (DATA)
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_4561.trc  (incident=55419):
ORA-00600: internal error code, arguments: [kfrValAcd30], [DATA], [1], [32], [568], [33], [569], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_55419/+ASM1_ora_4561_i55419.tr

1. ASM disk information before  I wipe them out.

[root@rac1 ~]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        2610    20860402+  8e  Linux LVM

Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1305    10482381   83  Linux

Disk /dev/sdc: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1        1305    10482381   83  Linux

Disk /dev/sdd: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1        1305    10482381   83  Linux

Disk /dev/sde: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1        1305    10482381   83  Linux

Disk /dev/sdf: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1        1305    10482381   83  Linux

Disk /dev/sdg: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdg1               1        1305    10482381   83  Linux

Disk /dev/sdh: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdh1               1        2610    20964793+  83  Linux

Disk /dev/sdi: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdi1               1        1044     8385898+  83  Linux

Disk /dev/dm-0: 17.1 GB, 17112760320 bytes
255 heads, 63 sectors/track, 2080 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 4227 MB, 4227858432 bytes
255 heads, 63 sectors/track, 514 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-1 doesn't contain a valid partition table

[root@rac1 ~]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
FRA

[root@rac1 ~]# oracleasm querydisk -p DISK1
Disk "DISK1" is a valid ASM disk
/dev/sdb1: LABEL="DISK1" TYPE="oracleasm"
[root@rac1 ~]# oracleasm querydisk -p DISK2
Disk "DISK2" is a valid ASM disk
/dev/sdc1: LABEL="DISK2" TYPE="oracleasm"
[root@rac1 ~]# oracleasm querydisk -p DISK3
Disk "DISK3" is a valid ASM disk
/dev/sdd1: LABEL="DISK3" TYPE="oracleasm"
[root@rac1 ~]# oracleasm querydisk -p DISK4
Disk "DISK4" is a valid ASM disk
/dev/sde1: LABEL="DISK4" TYPE="oracleasm"
[root@rac1 ~]# oracleasm querydisk -p DISK5
Disk "DISK5" is a valid ASM disk
/dev/sdf1: LABEL="DISK5" TYPE="oracleasm"

[root@rac1 ~]# oracleasm querydisk -p FRA
Disk "FRA" is a valid ASM disk
/dev/sdg1: LABEL="FRA" TYPE="oracleasm"

2. Recreate ASM disks, which belong to two diskgroups: FRA (one disk) and DATA(5 disks).

SQL> alter diskgroup fra mount;

Diskgroup altered.

SQL> select * from v$asm_diskgroup;

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
           0 DATA                                     0       4096
                   0 DISMOUNTED                  0          0           0
           0                       0              0             0
0.0.0.0.0
0.0.0.0.0                                                    N

           1 FRA                                    512       4096
             1048576 MOUNTED     EXTERN      10236       8533           0
        1703                       0           8533             0
11.2.0.0.0
10.1.0.0.0                                                   N



SQL> select path,group_number from  v$asm_disk;

PATH                           GROUP_NUMBER
------------------------------ ------------
ORCL:DISK1                                0
ORCL:DISK2                                0
ORCL:FRA                                  0
ORCL:DISK4                                0
ORCL:DISK5                                0
ORCL:DISK3                                0

6 rows selected.


SQL> drop diskgroup fra force including contents;

Diskgroup dropped.



SQL> select * from v$asm_diskgroup;

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
           0 DATA                                     0       4096
                   0 DISMOUNTED                  0          0           0
           0                       0              0             0
0.0.0.0.0
0.0.0.0.0                                                    N


SQL> drop diskgroup DATA force including contents;
drop diskgroup DATA force including contents
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15276: ASM diskgroup DATA has cluster voting files


SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Real Application Clusters and Automatic Storage Management options
[oracle@rac1 ~]$  oerr ora 15276]
[oracle@rac1 ~]$  oerr ora 15276
15276, 00000, "ASM diskgroup %s has cluster voting files"
// *Cause:  An attempt was made to drop a diskgroup that contained cluster
//          voting files.
// *Action: Move the cluster voting files out of the diskgroup and retry the
//          operation.



[root@rac1 ~]# oracleasm deletedisk  DISK1
Clearing disk header: failed
Unable to clear disk "DISK1"
[root@rac1 ~]# oracleasm deletedisk  DISK2
Clearing disk header: done
Dropping disk: done
[root@rac1 ~]# oracleasm deletedisk  DISK3
Clearing disk header: done
Dropping disk: done
[root@rac1 ~]# oracleasm deletedisk  DISK4
Clearing disk header: done
Dropping disk: done
[root@rac1 ~]# oracleasm deletedisk  DISK5
Clearing disk header: done
Dropping disk: done

[root@rac1 ~]# oracleasm createdisk  DISK2 /dev/sdc1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac1 ~]# oracleasm listdisks
DISK1
DISK2
FRA
[root@rac1 ~]# oracleasm createdisk  DISK3 /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# oracleasm createdisk  DISK4 /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# oracleasm createdisk  DISK5 /dev/sdf1
Writing disk header: done
Instantiating disk: done

[root@rac1 ~]# dd if=/dev/zero of=/dev/sdb1 bs=1024 count=4
4+0 records in
4+0 records out
4096 bytes (4.1 kB) copied, 4.3731e-05 seconds, 93.7 MB/s
[root@rac1 ~]# oracleasm deletedisk  DISK1
Disk "DISK1" defines an unmarked device
Dropping disk: done
[root@rac1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac1 ~]# oracleasm listdisks
DISK2
DISK3
DISK4
DISK5
FRA
[root@rac1 ~]# oracleasm createdisk  DISK1 /dev/sdb1
Unable to open device "/dev/sdb1": Device or resource busy

# and there are still HAS processes running.

[root@rac1 ~]# ps -ef |grep grid
root      3187     1  0 Mar20 ?        00:00:04 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
oracle    3813     1  0 Mar20 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle    3827     1  0 Mar20 ?        00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
oracle    3838     1  0 Mar20 ?        00:00:06 /u01/app/11.2.0/grid/bin/gpnpd.bin
oracle    3850     1  0 Mar20 ?        00:00:02 /u01/app/11.2.0/grid/bin/gipcd.bin
root      3916     1  0 Mar20 ?        00:00:00 /u01/app/11.2.0/grid/bin/cssdmonitor
root      3931     1  0 Mar20 ?        00:00:00 /u01/app/11.2.0/grid/bin/cssdagent
oracle    3982     1  0 Mar20 ?        00:00:10 /u01/app/11.2.0/grid/bin/ocssd.bin
root      3995     1  0 Mar20 ?        00:00:14 /u01/app/11.2.0/grid/bin/orarootagent.bin
root      4009     1  0 Mar20 ?        00:00:14 /u01/app/11.2.0/grid/bin/osysmond.bin
oracle    4011     1  0 Mar20 ?        00:00:00 /u01/app/11.2.0/grid/bin/diskmon.bin -d -f
root      4215     1  0 Mar20 ?        00:00:13 /u01/app/11.2.0/grid/bin/ologgerd -M -d /u01/app/11.2.0/grid/crf/db/rac1
root      4267     1  0 Mar20 ?        00:00:01 /u01/app/11.2.0/grid/bin/octssd.bin reboot
oracle    4291     1  0 Mar20 ?        00:00:00 /u01/app/11.2.0/grid/bin/evmd.bin
root     13782  8695  0 00:21 pts/1    00:00:00 grep grid
[root@rac1 ~]# . ~oracle/grid_env
[root@rac1 ~]# crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.



--after server reboot

[root@rac1 ~]# oracleasm createdisk  DISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@rac1 ~]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
FRA

3. below processes are auto-up and running. I have to stop them ...

[root@rac1 client]# ps -ef |grep grid
root      3187     1  0 23:11 ?        00:00:00 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
oracle    3811     1  0 23:11 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle    3825     1  0 23:11 ?        00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
oracle    3836     1  0 23:11 ?        00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin
oracle    3925     1  0 23:11 ?        00:00:00 /u01/app/11.2.0/grid/bin/gipcd.bin
root      4091     1  0 23:11 ?        00:00:00 /u01/app/11.2.0/grid/bin/orarootagent.bin
root      4120     1  0 23:11 ?        00:00:01 /u01/app/11.2.0/grid/bin/osysmond.bin
root      4346     1  0 23:21 ?        00:00:00 /u01/app/11.2.0/grid/bin/cssdmonitor
root      4360     1  0 23:21 ?        00:00:00 /u01/app/11.2.0/grid/bin/cssdagent
oracle    4374     1  0 23:21 ?        00:00:00 /u01/app/11.2.0/grid/bin/ocssd.bin

[root@rac1 init.d]# pwd
/etc/init.d
[root@rac1 init.d]# ls -l init*
-rwxr-xr-x 1 root root 8767 Mar 20  2011 init.ohasd
[root@rac1 init.d]# ./init.ohasd stop


[root@rac1 init.d]# crsctl stop crs
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.


[root@rac1 client]# ps -ef |grep oracle
root      4508  4300  0 23:32 pts/2    00:00:00 grep oracle


# now all process are gone. alternatively, can kill them.

4. deconfig clusterware services
[oracle@rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.2.0 Production on Tue Mar 26 23:47:27 2013

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount;
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service
SQL> exit
Disconnected


[root@rac1 install]# ./rootcrs.pl -deconfig
Using configuration parameter file: ./crsconfig_params
Oracle Clusterware stack is not active on this node
Restart the clusterware stack (use /u01/app/11.2.0/grid/bin/crsctl start crs) and retry
Failed to verify resources
[root@rac1 install]# ./rootcrs.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

ACFS-9200: Supported
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully deconfigured Oracle clusterware stack on this node

[root@rac1 install]# ./roothas.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Delete failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
You must kill ohasd processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
ACFS-9200: Supported
ACFS-9313: No ADVM/ACFS installation detected.
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Failure in execution (rc=-1, 256, No such file or directory) for command 1 /etc/init.d/ohasd deinstall
Successfully deconfigured Oracle Restart stack
[root@rac1 install]# ps -ef |grep has
root      5651  4233  0 00:11 pts/1    00:00:00 grep has

--finally run the root.sh to reconfigure clusterware services

[root@rac1 grid]# ./root.sh
Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Successful addition of voting disk 55c34b4c06194f09bf8018170dab7627.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   55c34b4c06194f09bf8018170dab7627 (ORCL:DISK1) [DATA]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'
CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded
ACFS-9200: Supported
ACFS-9200: Supported
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'
CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeeded
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

5. verification

[root@rac1 grid]# ps -ef |grep ASM
root      1825  4233  0 00:24 pts/1    00:00:00 grep ASM
oracle   13498     1  0 00:21 ?        00:00:00 asm_pmon_+ASM1
oracle   13504     1  0 00:21 ?        00:00:00 asm_psp0_+ASM1
oracle   13738     1  0 00:21 ?        00:00:00 asm_vktm_+ASM1
oracle   13762     1  0 00:21 ?        00:00:00 asm_gen0_+ASM1
oracle   13771     1  0 00:21 ?        00:00:00 asm_diag_+ASM1
oracle   13776     1  0 00:21 ?        00:00:00 asm_ping_+ASM1
oracle   13783     1  0 00:21 ?        00:00:00 asm_dia0_+ASM1
oracle   13790     1  0 00:21 ?        00:00:00 asm_lmon_+ASM1
oracle   13798     1  0 00:21 ?        00:00:00 asm_lmd0_+ASM1
oracle   13802     1  0 00:21 ?        00:00:00 asm_lms0_+ASM1
oracle   13811     1  0 00:21 ?        00:00:00 asm_lmhb_+ASM1
oracle   13820     1  0 00:21 ?        00:00:00 asm_mman_+ASM1
oracle   13830     1  0 00:21 ?        00:00:00 asm_dbw0_+ASM1
oracle   13838     1  0 00:21 ?        00:00:00 asm_lgwr_+ASM1
oracle   13843     1  0 00:21 ?        00:00:00 asm_ckpt_+ASM1
oracle   13852     1  0 00:21 ?        00:00:00 asm_smon_+ASM1
oracle   13860     1  0 00:21 ?        00:00:00 asm_rbal_+ASM1
oracle   13866     1  0 00:21 ?        00:00:00 asm_gmon_+ASM1
oracle   13872     1  0 00:21 ?        00:00:00 asm_mmon_+ASM1
oracle   13880     1  0 00:21 ?        00:00:00 asm_mmnl_+ASM1
oracle   13938     1  0 00:21 ?        00:00:00 asm_lck0_+ASM1
oracle   15325     1  0 00:21 ?        00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   15727     1  1 00:21 ?        00:00:01 oracle+ASM1_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   15733     1  0 00:21 ?        00:00:00 asm_asmb_+ASM1
oracle   15743     1  0 00:21 ?        00:00:00 oracle+ASM1_asmb_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   15754     1  0 00:21 ?        00:00:00 asm_o000_+ASM1
oracle   15762     1  0 00:21 ?        00:00:00 oracle+ASM1_o000_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   26710     1  0 00:22 ?        00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
[root@rac1 grid]# ps -ef |grep oracle
root      1827  4233  0 00:24 pts/1    00:00:00 grep oracle
root      4812  4300  0 Mar26 pts/2    00:00:00 su - oracle
oracle    4813  4812  0 Mar26 pts/2    00:00:00 -bash
oracle   13498     1  0 00:21 ?        00:00:00 asm_pmon_+ASM1
oracle   13504     1  0 00:21 ?        00:00:00 asm_psp0_+ASM1
oracle   13738     1  0 00:21 ?        00:00:00 asm_vktm_+ASM1
oracle   13762     1  0 00:21 ?        00:00:00 asm_gen0_+ASM1
oracle   13771     1  0 00:21 ?        00:00:00 asm_diag_+ASM1
oracle   13776     1  0 00:21 ?        00:00:00 asm_ping_+ASM1
oracle   13783     1  0 00:21 ?        00:00:00 asm_dia0_+ASM1
oracle   13790     1  0 00:21 ?        00:00:00 asm_lmon_+ASM1
oracle   13798     1  0 00:21 ?        00:00:00 asm_lmd0_+ASM1
oracle   13802     1  0 00:21 ?        00:00:00 asm_lms0_+ASM1
oracle   13811     1  0 00:21 ?        00:00:00 asm_lmhb_+ASM1
oracle   13820     1  0 00:21 ?        00:00:00 asm_mman_+ASM1
oracle   13830     1  0 00:21 ?        00:00:00 asm_dbw0_+ASM1
oracle   13838     1  0 00:21 ?        00:00:00 asm_lgwr_+ASM1
oracle   13843     1  0 00:21 ?        00:00:00 asm_ckpt_+ASM1
oracle   13852     1  0 00:21 ?        00:00:00 asm_smon_+ASM1
oracle   13860     1  0 00:21 ?        00:00:00 asm_rbal_+ASM1
oracle   13866     1  0 00:21 ?        00:00:00 asm_gmon_+ASM1
oracle   13872     1  0 00:21 ?        00:00:00 asm_mmon_+ASM1
oracle   13880     1  0 00:21 ?        00:00:00 asm_mmnl_+ASM1
oracle   13938     1  0 00:21 ?        00:00:00 asm_lck0_+ASM1
oracle   15325     1  0 00:21 ?        00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   15720     1  0 00:21 ?        00:00:00 /u01/app/11.2.0/grid/bin/evmd.bin
oracle   15727     1  0 00:21 ?        00:00:01 oracle+ASM1_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   15733     1  0 00:21 ?        00:00:00 asm_asmb_+ASM1
oracle   15743     1  0 00:21 ?        00:00:00 oracle+ASM1_asmb_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   15754     1  0 00:21 ?        00:00:00 asm_o000_+ASM1
oracle   15762     1  0 00:21 ?        00:00:00 oracle+ASM1_o000_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   16222 15720  0 00:21 ?        00:00:00 /u01/app/11.2.0/grid/bin/evmlogger.bin -o /u01/app/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/11.2.0/grid/evm/log/evmlogger.log
oracle   26509     1  0 00:22 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle   26710     1  0 00:22 ?        00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   30286     1  0 00:23 ?        00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
oracle   30287 30286  0 00:23 ?        00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
oracle   30801     1  0 00:20 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle   30840     1  0 00:20 ?        00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
oracle   31165     1  0 00:20 ?        00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin
oracle   31433     1  0 00:23 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
oracle   31558     1  0 00:20 ?        00:00:00 /u01/app/11.2.0/grid/bin/gipcd.bin
oracle   31851     1  0 00:20 ?        00:00:00 /u01/app/11.2.0/grid/bin/diskmon.bin -d -f
oracle   31923     1  0 00:20 ?        00:00:00 /u01/app/11.2.0/grid/bin/ocssd.bin
oracle   32011     1  0 00:23 ?        00:00:00 /u01/app/11.2.0/grid/bin/scriptagent.bin
oracle   32093     1  9 00:23 ?        00:00:07 /u01/app/11.2.0/grid/jdk/jre//bin/java -server -Xcheck:jni -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Dstdstream.filesize=100 -Dstdstream.filenumber=10 -DTRACING.ENABLED=false -Doracle.wlm.dbwlmlogger.logging.level=INFO -Dport.rmi=23792 -jar /u01/app/11.2.0/grid/oc4j/j2ee/home/oc4j.jar -config /u01/app/11.2.0/grid/oc4j/j2ee/home/OC4J_DBWLM_config/server.xml -out /u01/app/11.2.0/grid/oc4j/j2ee/home/log/oc4j.out -err /u01/app/11.2.0/grid/oc4j/j2ee/home/log/oc4j.err

SQL> set pages 1000
SQL> select * from v$asm_diskgroup;

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
           1 DATA                                   512       4096
             1048576 MOUNTED     EXTERN      51180      50776           0
         404                       0          50776             0
11.2.0.0.0
10.1.0.0.0                                                   Y


6. do the same in node2

[root@rac2 install]# ls -l root*
-rwxr-xr-x 1 oracle oinstall 29259 Mar 20  2011 rootcrs.pl
-rwxr-xr-x 1 oracle oinstall 13522 Mar 20  2011 roothas.pl
-rwxr-xr-x 1 oracle oinstall   915 Mar 20  2011 rootofs.sh
[root@rac2 install]# ./roothas.pl -deconfig
Using configuration parameter file: ./crsconfig_params
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
You must kill ohasd processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
ACFS-9200: Supported
Successfully deconfigured Oracle Restart stack
[root@rac2 install]# ./rootcrs.pl -deconfig
Using configuration parameter file: ./crsconfig_params
Oracle Clusterware stack is not active on this node
Restart the clusterware stack (use /u01/app/11.2.0/grid/bin/crsctl start crs) and retry
Failed to verify resources



[root@rac2 install]# which oracleasm
/usr/sbin/oracleasm

[root@rac2 install]#  oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac2 install]#  oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
FRA
[root@rac2 install]# cd ../..
[root@rac2 grid]# pwd
/u01/app/11.2.0/grid

[root@rac2 grid]# ./root.sh
Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[root@rac2 grid]# ps -ef |grep pmon
oracle    8105     1  0 00:30 ?        00:00:00 asm_pmon_+ASM2
root      9282  4030  0 00:40 pts/1    00:00:00 grep pmon

[root@rac2 grid]# ps -ef |grep ASM2
oracle    8105     1  0 00:30 ?        00:00:00 asm_pmon_+ASM2
oracle    8109     1  0 00:30 ?        00:00:00 asm_psp0_+ASM2
oracle    8113     1  0 00:30 ?        00:00:00 asm_vktm_+ASM2
oracle    8119     1  0 00:30 ?        00:00:00 asm_gen0_+ASM2
oracle    8123     1  0 00:30 ?        00:00:00 asm_diag_+ASM2
oracle    8127     1  0 00:30 ?        00:00:00 asm_ping_+ASM2
oracle    8131     1  0 00:30 ?        00:00:00 asm_dia0_+ASM2
oracle    8135     1  0 00:30 ?        00:00:00 asm_lmon_+ASM2
oracle    8139     1  0 00:30 ?        00:00:00 asm_lmd0_+ASM2
oracle    8143     1  0 00:30 ?        00:00:00 asm_lms0_+ASM2
oracle    8149     1  0 00:30 ?        00:00:00 asm_lmhb_+ASM2
oracle    8153     1  0 00:30 ?        00:00:00 asm_mman_+ASM2
oracle    8157     1  0 00:30 ?        00:00:00 asm_dbw0_+ASM2
oracle    8161     1  0 00:30 ?        00:00:00 asm_lgwr_+ASM2
oracle    8165     1  0 00:30 ?        00:00:00 asm_ckpt_+ASM2
oracle    8169     1  0 00:30 ?        00:00:00 asm_smon_+ASM2
oracle    8173     1  0 00:30 ?        00:00:00 asm_rbal_+ASM2
oracle    8177     1  0 00:30 ?        00:00:00 asm_gmon_+ASM2
oracle    8181     1  0 00:30 ?        00:00:00 asm_mmon_+ASM2
oracle    8185     1  0 00:30 ?        00:00:00 asm_mmnl_+ASM2
oracle    8189     1  0 00:31 ?        00:00:00 asm_lck0_+ASM2
oracle    8206     1  0 00:31 ?        00:00:00 oracle+ASM2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    8239     1  0 00:31 ?        00:00:00 oracle+ASM2_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    8248     1  0 00:31 ?        00:00:00 asm_asmb_+ASM2
oracle    8252     1  0 00:31 ?        00:00:00 oracle+ASM2_asmb_+asm2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    8408     1  0 00:31 ?        00:00:00 oracle+ASM2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    9221     1  0 00:39 ?        00:00:00 asm_gcr0_+ASM2
root      9286  4030  0 00:40 pts/1    00:00:00 grep ASM2

[root@rac2 grid]# ps -ef |grep oracle
oracle    7800     1  0 00:30 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle    7814     1  0 00:30 ?        00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
oracle    7830     1  0 00:30 ?        00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin
oracle    7855     1  0 00:30 ?        00:00:00 /u01/app/11.2.0/grid/bin/gipcd.bin
oracle    7907     1  0 00:30 ?        00:00:01 /u01/app/11.2.0/grid/bin/ocssd.bin
oracle    7910     1  0 00:30 ?        00:00:00 /u01/app/11.2.0/grid/bin/diskmon.bin -d -f
oracle    8105     1  0 00:30 ?        00:00:00 asm_pmon_+ASM2
oracle    8109     1  0 00:30 ?        00:00:00 asm_psp0_+ASM2
oracle    8113     1  0 00:30 ?        00:00:00 asm_vktm_+ASM2
oracle    8119     1  0 00:30 ?        00:00:00 asm_gen0_+ASM2
oracle    8123     1  0 00:30 ?        00:00:00 asm_diag_+ASM2
oracle    8127     1  0 00:30 ?        00:00:00 asm_ping_+ASM2
oracle    8131     1  0 00:30 ?        00:00:00 asm_dia0_+ASM2
oracle    8135     1  0 00:30 ?        00:00:00 asm_lmon_+ASM2
oracle    8139     1  0 00:30 ?        00:00:00 asm_lmd0_+ASM2
oracle    8143     1  0 00:30 ?        00:00:00 asm_lms0_+ASM2
oracle    8149     1  0 00:30 ?        00:00:00 asm_lmhb_+ASM2
oracle    8153     1  0 00:30 ?        00:00:00 asm_mman_+ASM2
oracle    8157     1  0 00:30 ?        00:00:00 asm_dbw0_+ASM2
oracle    8161     1  0 00:30 ?        00:00:00 asm_lgwr_+ASM2
oracle    8165     1  0 00:30 ?        00:00:00 asm_ckpt_+ASM2
oracle    8169     1  0 00:30 ?        00:00:00 asm_smon_+ASM2
oracle    8173     1  0 00:30 ?        00:00:00 asm_rbal_+ASM2
oracle    8177     1  0 00:30 ?        00:00:00 asm_gmon_+ASM2
oracle    8181     1  0 00:30 ?        00:00:00 asm_mmon_+ASM2
oracle    8185     1  0 00:30 ?        00:00:00 asm_mmnl_+ASM2
oracle    8189     1  0 00:31 ?        00:00:00 asm_lck0_+ASM2
oracle    8206     1  0 00:31 ?        00:00:00 oracle+ASM2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    8236     1  0 00:31 ?        00:00:00 /u01/app/11.2.0/grid/bin/evmd.bin
oracle    8239     1  0 00:31 ?        00:00:00 oracle+ASM2_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    8248     1  0 00:31 ?        00:00:00 asm_asmb_+ASM2
oracle    8252     1  0 00:31 ?        00:00:00 oracle+ASM2_asmb_+asm2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    8326  8236  0 00:31 ?        00:00:00 /u01/app/11.2.0/grid/bin/evmlogger.bin -o /u01/app/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/11.2.0/grid/evm/log/evmlogger.log
oracle    8367     1  0 00:31 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle    8408     1  0 00:31 ?        00:00:00 oracle+ASM2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    8431     1  0 00:31 ?        00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
oracle    8432  8431  0 00:31 ?        00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
oracle    9221     1  0 00:39 ?        00:00:00 asm_gcr0_+ASM2
root      9289  4030  0 00:40 pts/1    00:00:00 grep oracle



SQL> l
  1* select * from v$asm_diskgroup
SQL> /

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB
-------------------- ----------- ------ ---------- ---------- -----------
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
------------ ----------------------- -------------- -------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY                                       V
------------------------------------------------------------ -
           1 DATA                                   512       4096
             1048576 MOUNTED     EXTERN      51180      50776           0
         404                       0          50776             0
11.2.0.0.0
10.1.0.0.0


References:
http://hiteshgondalia.wordpress.com/2013/03/10/crs-4046-invalid-oracle-clusterware-configuration-11gr2/

test recovery tablespace command

RMAN> sql 'alter tablespace tts1 offline';

sql statement: alter tablespace tts1 offline

RMAN> sql 'alter tablespace tts1 online';

sql statement: alter tablespace tts1 online

RMAN> sql 'alter tablespace tts1 offline immediate';

sql statement: alter tablespace tts1 offline immediate

RMAN> sql 'alter tablespace tts1 online';

sql statement: alter tablespace tts1 online
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of sql command on default channel at 03/28/2013 14:02:55
RMAN-11003: failure during parse/execution of SQL statement: alter tablespace tts1 online
ORA-01113: file 5 needs media recovery
ORA-01110: data file 5: '/u02/oradata/TTS/tts1_01.bdf'

RMAN> recover tablespace tts1 ;

Starting recover at Mar 28 2013 14:03:11
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at Mar 28 2013 14:03:12

RMAN> sql 'alter tablespace tts1 online';

sql statement: alter tablespace tts1 online


A more likely scenario in real world is particular disk/filesystem is not available, thus we need to relocate those affected data files.

1. backup the tablespace TTS1

RMAN> report schema;

Report of database schema for database with db_unique_name TTS

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    700      SYSTEM               ***     /u02/oradata/TTS/system01.dbf
2    600      SYSAUX               ***     /u02/oradata/TTS/sysaux01.dbf
3    370      UNDOTBS1             ***     /u02/oradata/TTS/undotbs01.dbf
4    5        USERS                ***     /u02/oradata/TTS/users01.dbf
5    20       TTS1                 ***     /u02/oradata/TTS/tts1_01.bdf

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    20       TEMP                 32767       /u02/oradata/TTS/temp01.dbf



RMAN> backup tablespace tts1;

Starting backup at Mar 28 2013 14:29:58
using channel ORA_DISK_1
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00005 name=/u02/oradata/TTS/tts1_01.bdf
channel ORA_DISK_1: starting piece 1 at Mar 28 2013 14:29:59
channel ORA_DISK_1: finished piece 1 at Mar 28 2013 14:30:00
piece handle=/u02/oradata/TTS/FRA/TTS/backupset/2013_03_28/o1_mf_nnndf_TAG201303                             28T142958_8o7rv78f_.bkp tag=TAG20130328T142958 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at Mar 28 2013 14:30:00

Starting Control File and SPFILE Autobackup at Mar 28 2013 14:30:00
piece handle=/u02/oradata/TTS/FRA/TTS/autobackup/2013_03_28/o1_mf_s_811261800_8o                             7rv8gp_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at Mar 28 2013 14:30:01



RMAN> list backup of tablespace tts1;


List of Backup Sets
===================


BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
24      Incr 0  204.51M    DISK        00:01:20     Mar 27 2013 15:27:08
        BP Key: 24   Status: AVAILABLE  Compressed: YES  Tag: TAG20130327T152547
        Piece Name: /u02/oradata/TTS/FRA/TTS/backupset/2013_03_27/o1_mf_nnnd0_TAG20130327T152547_8o57qx5l_.bkp
  List of Datafiles in backup set 24
  File LV Type Ckp SCN    Ckp Time             Name
  ---- -- ---- ---------- -------------------- ----
  5    0  Incr 1021037    Mar 27 2013 15:25:49 /u02/oradata/TTS/tts1_01.bdf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
28      Incr 1  312.00K    DISK        00:00:31     Mar 27 2013 15:32:29
        BP Key: 28   Status: AVAILABLE  Compressed: YES  Tag: TAG20130327T153155
        Piece Name: /u02/oradata/TTS/FRA/TTS/backupset/2013_03_27/o1_mf_nnnd1_TAG20130327T153155_8o583gsl_.bkp
  List of Datafiles in backup set 28
  File LV Type Ckp SCN    Ckp Time             Name
  ---- -- ---- ---------- -------------------- ----
  5    1  Incr 1021321    Mar 27 2013 15:31:58 /u02/oradata/TTS/tts1_01.bdf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
33      Full    2.05M      DISK        00:00:01     Mar 28 2013 14:30:00
        BP Key: 33   Status: AVAILABLE  Compressed: YES  Tag: TAG20130328T142958
        Piece Name: /u02/oradata/TTS/FRA/TTS/backupset/2013_03_28/o1_mf_nnndf_TAG20130328T142958_8o7rv78f_.bkp
  List of Datafiles in backup set 33
  File LV Type Ckp SCN    Ckp Time             Name
  ---- -- ---- ---------- -------------------- ----
  5       Full 1101322    Mar 28 2013 14:29:59 /u02/oradata/TTS/tts1_01.bdf

2. Simulate the disk is gone.

RMAN> sql 'alter tablespace tts1 offline immediate';

sql statement: alter tablespace tts1 offline immediate

RMAN> set newname for datafile '/u02/oradata/TTS/tts1_01.bdf' to '/home/oracle/tts1_01.dbf';

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03031: this option of set command needs to be used inside a run block

RMAN> run {
2> set newname for datafile '/u02/oradata/TTS/tts1_01.bdf' to '/home/oracle/tts1_01.dbf';
3> restore tablespace tts1;
4> switch datafile all;
5> recover tablespace tts1;
6> sql 'alter tablespace tts1 online ' ;
7> }

executing command: SET NEWNAME

Starting restore at Mar 28 2013 14:33:37
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00005 to /home/oracle/tts1_01.dbf
channel ORA_DISK_1: reading from backup piece /u02/oradata/TTS/FRA/TTS/backupset/2013_03_28/o1_mf_nnndf_TAG20130328T142958_8o7rv78f_.bkp
channel ORA_DISK_1: piece handle=/u02/oradata/TTS/FRA/TTS/backupset/2013_03_28/o1_mf_nnndf_TAG20130328T142958_8o7rv78f_.bkp tag=TAG20130328T142958
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
Finished restore at Mar 28 2013 14:33:38

datafile 5 switched to datafile copy
input datafile copy RECID=24 STAMP=811262018 file name=/home/oracle/tts1_01.dbf

Starting recover at Mar 28 2013 14:33:38
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at Mar 28 2013 14:33:38

sql statement: alter tablespace tts1 online

RMAN> report schema;

Report of database schema for database with db_unique_name TTS

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    700      SYSTEM               ***     /u02/oradata/TTS/system01.dbf
2    600      SYSAUX               ***     /u02/oradata/TTS/sysaux01.dbf
3    370      UNDOTBS1             ***     /u02/oradata/TTS/undotbs01.dbf
4    5        USERS                ***     /u02/oradata/TTS/users01.dbf
5    20       TTS1                 ***     /home/oracle/tts1_01.dbf

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    20       TEMP                 32767       /u02/oradata/TTS/temp01.dbf


3. after disaster,  put the datafile to original location;


RMAN> backup as copy datafile 5 format '/u02/oradata/TTS/tts1_01.bdf';

Starting backup at Mar 28 2013 14:37:41
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=/home/oracle/tts1_01.dbf
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/28/2013 14:37:42
ORA-19504: failed to create file "/u02/oradata/TTS/tts1_01.bdf"
ORA-27038: created file already exists
Additional information: 1

RMAN> host ' rm /u02/oradata/TTS/tts1_01.bdf ';

host command complete

RMAN> backup as copy datafile 5 format '/u02/oradata/TTS/tts1_01.bdf';

Starting backup at Mar 28 2013 14:37:59
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=/home/oracle/tts1_01.dbf
output file name=/u02/oradata/TTS/tts1_01.bdf tag=TAG20130328T143759 RECID=26 STAMP=811262280
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at Mar 28 2013 14:38:00

Starting Control File and SPFILE Autobackup at Mar 28 2013 14:38:00
piece handle=/u02/oradata/TTS/FRA/TTS/autobackup/2013_03_28/o1_mf_s_811262280_8o7sb92t_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at Mar 28 2013 14:38:01

RMAN> sql 'alter database datafile 5 offline ' ;

sql statement: alter database datafile 5 offline


RMAN> switch datafile 5 to copy;

datafile 5 switched to datafile copy "/u02/oradata/TTS/tts1_01.bdf"

RMAN> sql 'alter database datafile 5 online ' ;

sql statement: alter database datafile 5 online
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of sql command on default channel at 03/28/2013 14:40:30
RMAN-11003: failure during parse/execution of SQL statement: alter database datafile 5 online
ORA-01113: file 5 needs media recovery
ORA-01110: data file 5: '/u02/oradata/TTS/tts1_01.bdf'

RMAN> recover datafile 5  ;

Starting recover at Mar 28 2013 14:40:47
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at Mar 28 2013 14:40:47

RMAN> sql 'alter database datafile 5 online ' ;

sql statement: alter database datafile 5 online

RMAN> report schema;

Report of database schema for database with db_unique_name TTS

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    700      SYSTEM               ***     /u02/oradata/TTS/system01.dbf
2    600      SYSAUX               ***     /u02/oradata/TTS/sysaux01.dbf
3    370      UNDOTBS1             ***     /u02/oradata/TTS/undotbs01.dbf
4    5        USERS                ***     /u02/oradata/TTS/users01.dbf
5    20       TTS1                 ***     /u02/oradata/TTS/tts1_01.bdf

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    20       TEMP                 32767       /u02/oradata/TTS/temp01.dbf

RMAN> exit


Recovery Manager complete.

4. Verification

orarac2poc:TTS:/home/oracle> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Thu Mar 28 14:42:10 2013

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options


SYS@TTS> select table_name ,owner from dba_TABLEs where tablespace_name='TTS1';

TABLE_NAME                     OWNER
------------------------------ ------------------------------
T01                            TRANP
T02                            TRANP
IDX03                          TRANP

SYS@TTS> select count(*) from tranp.t01;

  COUNT(*)
----------
     65949

Flashback database

Objectives:
a.)  Do we need an image copy of datafiles in Fast Recovery Area?
b.)  Can flashback database run in NOARCHIVELOG mode?
So that I can estimated least space needed for project development databases.

1. ensure no image copy in FRA

orarac2poc:TTS:/home/oracle> rman target /

Recovery Manager: Release 11.2.0.2.0 - Production on Wed Mar 27 09:21:07 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: TTS (DBID=1769654909)

RMAN> list backup;

using target database control file instead of recovery catalog
specification does not match any backup in the repository

RMAN> list copy;

specification does not match any datafile copy in the repository
specification does not match any control file copy in the repository
specification does not match any archived log in the repository

RMAN> exit


Recovery Manager complete.

2. Note down the SCN and enable flashback database

orarac2poc:TTS:/home/oracle> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Wed Mar 27 09:21:54 2013

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options


SYS@TTS> select flashback_on from v$database;

FLASHBACK_ON
------------------
NO

SYS@TTS> show parameter db_recovery

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string
db_recovery_file_dest_size           big integer 0
SYS@TTS> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SYS@TTS> startup mount
ORACLE instance started.

Total System Global Area  663908352 bytes
Fixed Size                  2229440 bytes
Variable Size             281021248 bytes
Database Buffers          373293056 bytes
Redo Buffers                7364608 bytes
Database mounted.
SYS@TTS> alter system set db_recovery_file_dest='/u02/oradata/TTS/FRA' ;
alter system set db_recovery_file_dest='/u02/oradata/TTS/FRA'
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-01261: Parameter db_recovery_file_dest destination string cannot be translated
ORA-01262: Stat failed on a file destination directory
Linux-x86_64 Error: 2: No such file or directory

--due to the subdir FRA was not pre-created. created it accordingly in other session.

SYS@TTS> alter system set db_recovery_file_dest='/u02/oradata/TTS/FRA' ;
alter system set db_recovery_file_dest='/u02/oradata/TTS/FRA'
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-19802: cannot use DB_RECOVERY_FILE_DEST without DB_RECOVERY_FILE_DEST_SIZE


SYS@TTS> alter system set db_recovery_file_dest_size=3000m ;

System altered.

SYS@TTS> alter system set db_recovery_file_dest='/u02/oradata/TTS/FRA' ;

System altered.

SYS@TTS> show parameter flashback

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_flashback_retention_target        integer     1440

SYS@TTS> select 1440/24 from dual;

   1440/24
----------
        60
       
#default 24 hours

SYS@TTS> alter database flashback on;

Database altered.

SYS@TTS> alter database open;

Database altered.


SYS@TTS> set time on
09:34:27 SYS@TTS> select count(*) from tranp.idx03;

  COUNT(*)
----------
      3491

09:34:38 SYS@TTS> select current_scn from v$database;

CURRENT_SCN
-----------
    1009053

09:34:43 SYS@TTS>
09:34:56 SYS@TTS>
09:34:58 SYS@TTS>
09:35:00 SYS@TTS> select current_scn from v$database;

CURRENT_SCN
-----------
    1009061

09:35:03 SYS@TTS> drop table tranp.idx03;

Table dropped.

09:35:21 SYS@TTS> select current_scn from v$database;

CURRENT_SCN
-----------
    1009099

09:35:43 SYS@TTS> alter system checkpoint;

System altered.

09:35:48 SYS@TTS> desc v$flashback_database_log;
 Name                                                                     Null?    Type
 ------------------------------------------------------------------------ -------- -------------------------------------------------
 OLDEST_FLASHBACK_SCN                                                              NUMBER
 OLDEST_FLASHBACK_TIME                                                             DATE
 RETENTION_TARGET                                                                  NUMBER
 FLASHBACK_SIZE                                                                    NUMBER
 ESTIMATED_FLASHBACK_SIZE                                                          NUMBER


09:36:41 SYS@TTS> @/home/oracle/scripts/performance/set_df

Session altered.

09:37:05 SYS@TTS> select * from v$flashback_database_log;

OLDEST_FLASHBACK_SCN OLDEST_FLASHBACK_TIM RETENTION_TARGET FLASHBACK_SIZE ESTIMATED_FLASHBACK_SIZE
-------------------- -------------------- ---------------- -------------- ------------------------
             1008730 27-mar-2013 09:33:19             1440       16384000                        0




09:38:38 SYS@TTS> select count(*) from tranp.idx03;
select count(*) from tranp.idx03
                           *
ERROR at line 1:
ORA-00942: table or view does not exist


3. Flashback database and verify

09:38:45 SYS@TTS> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
09:38:59 SYS@TTS> startup mount
ORACLE instance started.

Total System Global Area  663908352 bytes
Fixed Size                  2229440 bytes
Variable Size             281021248 bytes
Database Buffers          373293056 bytes
Redo Buffers                7364608 bytes
Database mounted.
09:39:27 SYS@TTS> select current_scn from v$database;

CURRENT_SCN
-----------
          0

09:39:47 SYS@TTS> flashback database to scn 1009053;

Flashback complete.

09:40:39 SYS@TTS> alter database open read only;

Database altered.

09:41:13 SYS@TTS> select count(*) from tranp.idx03;

  COUNT(*)
----------
      3491

09:42:10 SYS@TTS> select flashback_on from v$database;

FLASHBACK_ON
------------------
YES

09:43:04 SYS@TTS> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01531: a database already open by the instance


09:43:29 SYS@TTS> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
09:43:43 SYS@TTS> startup mount
ORACLE instance started.

Total System Global Area  663908352 bytes
Fixed Size                  2229440 bytes
Variable Size             281021248 bytes
Database Buffers          373293056 bytes
Redo Buffers                7364608 bytes
Database mounted.
09:44:06 SYS@TTS> alter database open resetlogs;

Database altered.

09:44:20 SYS@TTS> select count(*) from tranp.idx03;

  COUNT(*)
----------
      3491



orarac2poc:DGS:/u02/oradata/TTS/FRA/TTS/flashback> ls -l
total 16040
-rw-r----- 1 oracle asmadmin 8200192 Mar 27 09:39 o1_mf_8o4m2z8r_.flb
-rw-r----- 1 oracle asmadmin 8200192 Mar 27 09:33 o1_mf_8o4m2zm9_.flb
orarac2poc:DGS:/u02/oradata/TTS/FRA/TTS/flashback> ls -l
total 16040
-rw-r----- 1 oracle asmadmin 8200192 Mar 27 09:40 o1_mf_8o4m2z8r_.flb
-rw-r----- 1 oracle asmadmin 8200192 Mar 27 09:33 o1_mf_8o4m2zm9_.flb


4. check relationship with ARCHIVELOG mode

10:17:29 SYS@TTS> select log_mode from v$database;

LOG_MODE
------------
ARCHIVELOG

10:17:36 SYS@TTS> show parameter archive_log
10:17:51 SYS@TTS> show parameter log_archive_dest

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest                     string
log_archive_dest_1                   string
...

10:18:01 SYS@TTS> alter system set log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST';

System altered.

10:18:57 SYS@TTS> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
10:19:25 SYS@TTS> startup mount
ORACLE instance started.

Total System Global Area  663908352 bytes
Fixed Size                  2229440 bytes
Variable Size             281021248 bytes
Database Buffers          373293056 bytes
Redo Buffers                7364608 bytes
Database mounted.
10:21:36 SYS@TTS> alter database noarchivelog;
alter database noarchivelog
*
ERROR at line 1:
ORA-38774: cannot disable media recovery - flashback database is enabled


10:21:44 SYS@TTS> alter database open;

Database altered.

10:22:07 SYS@TTS> alter system switch logfile;

System altered.


Findings:

1. No image copy is required in order to use flashback database.
2. "The database must be running in ARCHIVELOG mode.
   -- To rewind the database to a guaranteed restore point, the FLASHBACK DATABASE command needs the archived redo logs starting from around the time of the restore point.
   -- A Fast Recovery Area must be configured because the Oracle Database server stores the required logs in the Fast Recovery Area.


Note that we cannot use Flashback Database in the following situations:
– The control file has been restored or re-created.
– A tablespace has been dropped.
– A data file has been reduced in size.

Delete backupset irregardless of deletion policy

1. files before delettion

orarac2poc:TTS:/u01/app/oracle/product/11.2.0/db_2/dbs> ls -l 01o536r8_1_1
-rw-r----- 1 oracle asmadmin 200278016 Mar 21 13:50 01o536r8_1_1
orarac2poc:TTS:/u01/app/oracle/product/11.2.0/db_2/dbs> ls -l c-1769654909-20130326-00
-rw-r----- 1 oracle asmadmin 9830400 Mar 26 15:05 c-1769654909-20130326-00

2. list backup sets

RMAN> report obsolete;

RMAN retention policy will be applied to the command
RMAN retention policy is set to redundancy 1
no obsolete backups found

RMAN> list backup ;


List of Backup Sets
===================


BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
1       Full    190.99M    DISK        00:01:11     21-MAR-13
        BP Key: 1   Status: AVAILABLE  Compressed: YES  Tag: TAG20130321T134928
        Piece Name: /u01/app/oracle/product/11.2.0/db_2/dbs/01o536r8_1_1
  List of Datafiles in backup set 1
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  1       Full 721426     21-MAR-13 /u02/oradata/TTS/system01.dbf
  2       Full 721426     21-MAR-13 /u02/oradata/TTS/sysaux01.dbf
  3       Full 721426     21-MAR-13 /u02/oradata/TTS/undotbs01.dbf
  4       Full 721426     21-MAR-13 /u02/oradata/TTS/users01.dbf
  5       Full 714646     21-MAR-13 /u02/oradata/TTS/tts1_01.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
2       Full    9.36M      DISK        00:00:02     26-MAR-13
        BP Key: 2   Status: AVAILABLE  Compressed: NO  Tag: TAG20130326T150504
        Piece Name: /u01/app/oracle/product/11.2.0/db_2/dbs/c-1769654909-20130326-00
  SPFILE Included: Modification time: 26-MAR-13
  SPFILE db_unique_name: TTS
  Control File Included: Ckp SCN: 976753       Ckp time: 26-MAR-13

3. Delete and verification
RMAN> delete backupset;

using channel ORA_DISK_1

List of Backup Pieces
BP Key  BS Key  Pc# Cp# Status      Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
1       1       1   1   AVAILABLE   DISK        /u01/app/oracle/product/11.2.0/db_2/dbs/01o536r8_1_1
2       2       1   1   AVAILABLE   DISK        /u01/app/oracle/product/11.2.0/db_2/dbs/c-1769654909-20130326-00

Do you really want to delete the above objects (enter YES or NO)? yes
deleted backup piece
backup piece handle=/u01/app/oracle/product/11.2.0/db_2/dbs/01o536r8_1_1 RECID=1 STAMP=810654568
deleted backup piece
backup piece handle=/u01/app/oracle/product/11.2.0/db_2/dbs/c-1769654909-20130326-00 RECID=2 STAMP=811091105
Deleted 2 objects


RMAN> list backup;

specification does not match any backup in the repository



orarac2poc:TTS:/u01/app/oracle/product/11.2.0/db_2/dbs> ls -l c-1769654909-20130326-00
ls: c-1769654909-20130326-00: No such file or directory
orarac2poc:TTS:/u01/app/oracle/product/11.2.0/db_2/dbs> ls -l 01o536r8_1_1
ls: 01o536r8_1_1: No such file or directory