Saturday, June 09, 2007

Waiting for smon to disable tx recovery

Yesterday when I was dropping a stalled big mview log (7gb, commad run for about 30mins ), system was rebooted by system administrator.

I don't what will happend to this database.

This morning, I tried to bring it up. luckily , it open successfully as usual.
Verified that :
--the mview log is not shown in dba_mview_logs
--the segment type is shown as TEMPORARY in dba_segments

I have no idea on this TEMPORARY segment, thought it okay as it is not production environment. We can solve it easily by sync it again.


However, when I tried to "shutdown immediate"
The DB looks DB is hanging at "shutdown immedaite" step. DB closed is not shown immediate as usual.
After 15 mins, I checked alert.log that the last line is:
"Waiting for smon to disable tx recovery." and the CPU utilization of smon is almost 100% . I realized it is doing some recovery. This could be related to the TEMPORARY segment, hope this recovery can help to clean it.

So I just let it run, no rush to open another session to issue "shutdown abort".

As the same time, I did some research on the internet. Which supports my suspect -- the DB is doing cleansing.

The relevant metalink notes is 1076161.6
"Verify that temporary segments are decreasing
---------------------------------------------
To verify that the temporary segments are decreasing have an active session
available in Server Manager during the SHUTDOWN IMMEDIATE. Issue the following
query to ensure the database is not hanging, but is actually perform extent
cleanup:

SVRMGR> select count(block#) from fet$;
COUNT(BLOC
----------
7

SVRMGR> select count(block#) from uet$;
COUNT(BLOC
----------
402

After some time has elapsed, reissue the query and see that the values for fet$
have increased while the values or uet$ have decreased:

SVRMGR> select count(block#) from fet$;
COUNT(BLOC
----------
10

SVRMGR> select count(block#) from uet$;
COUNT(BLOC
----------
399

During shutdown the SMON process is cleaning up extents and updating the data
dictionary tables with the marked free extents. As the extents are marked as
freed, they are removed from the table for used extents, UET$ and placed on the
table for free extents, FET$."


Finally, after 1.5 hours, the DB was shutdown gracefully.

Open it again, checked that the segment is no more there ,7Gb tablespace reclaimed!
Shutdown it within few seconds.

Attache the alert.log for your reference.
Fri Jun 8 09:16:03 2007
ARC1: Completed archiving log 4 thread 1 sequence 28
Fri Jun 8 09:26:18 2007
Shutting down instance: further logons disabled
Shutting down instance (immediate)
License high water mark = 4
Fri Jun 8 09:26:18 2007
ALTER DATABASE CLOSE NORMAL
Fri Jun 8 09:31:21 2007
Waiting for smon to disable tx recovery.
Fri Jun 8 11:05:07 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Fri Jun 8 11:05:08 2007
Shutting down archive processes
Archiving is disabled
Fri Jun 8 11:05:08 2007
ARCH shutting down
ARC1: Archival stopped
Fri Jun 8 11:05:08 2007
ARCH shutting down
Fri Jun 8 11:05:08 2007
ARC0: Archival stopped
Fri Jun 8 11:05:08 2007
Thread 1 closed at log sequence 29
Successful close of redo thread 1
Fri Jun 8 11:05:08 2007
Completed: ALTER DATABASE CLOSE NORMAL
Fri Jun 8 11:05:08 2007
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active