Powered By Blogger

Monday, May 23, 2011

Rolling forward the standby database by using primary database incremental backups

Scenario :

The archive gap is noted in the standby database alertlog as below. The required archivelog for recovery is not found in the primary and is been deleted in both the primary and standby unfortunately.

In this case the below is the recovery needed to synchronize the standby with primary. This incremental recovery only works from 10g onwards.

----------------------------------------------------------
Mon May 23 14:32:43 2011
FAL[client]: Failed to request gap sequence
 GAP - thread 1 sequence 174-174
 DBID 931617234 branch 747232149
FAL[client]: All defined FAL servers have been attempted.
-------------------------------------------------------------

Check the current SCN in the standby, that will be needed to take backup from that SC Number.

---- standby ----

SQL> select distinct checkpoint_change# from v$datafile_header order by 1;

CHECKPOINT_CHANGE#
------------------
            739339


Take BACKUP of primary database from scn 739339 as below

---- primary ----


D:\oracle\product\10.2.0\db_1\BIN>rman target 'sys/****** as sysdba'

Recovery Manager: Release 10.2.0.1.0 - Production on Mon May 23 14:44:30 2011

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: ORCL10G (DBID=931617234)

RMAN> backup incremental from scn 739339 database format 'd:\db\db_%U';

Starting backup at 23-MAY-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=141 devtype=DISK
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00004 name=D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\USERS01.DBF
input datafile fno=00001 name=D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSTEM01.DBF
input datafile fno=00002 name=D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\UNDOTBS01.DBF
input datafile fno=00003 name=D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSAUX01.DBF
channel ORA_DISK_1: starting piece 1 at 23-MAY-11
channel ORA_DISK_1: finished piece 1 at 23-MAY-11
piece handle=D:\DB\DB_0OMD2B30_1_1 tag=TAG20110523T144447 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:02:15
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
including current control file in backupset
including current SPFILE in backupset
channel ORA_DISK_1: starting piece 1 at 23-MAY-11
channel ORA_DISK_1: finished piece 1 at 23-MAY-11
piece handle=D:\DB\DB_0PMD2B77_1_1 tag=TAG20110523T144447 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 23-MAY-11

RMAN>


then copy the backup pieces to the standby server and catalog it and recover as below.


RMAN> catalog start with 'd:\db\';

searching for all files that match the pattern d:\db\

List of Files Unknown to the Database
=====================================
File Name: D:\DB\DB_0OMD2B30_1_1
File Name: D:\DB\DB_0PMD2B77_1_1

Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: D:\DB\DB_0OMD2B30_1_1
File Name: D:\DB\DB_0PMD2B77_1_1

RMAN> recover database noredo;

Starting recover at 23-MAY-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting incremental datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: D:\ORACLE\PRODUCT\10.2.0\ORADATA\STDBY10G\SYSTEM01.DBF
destination for restore of datafile 00002: D:\ORACLE\PRODUCT\10.2.0\ORADATA\STDBY10G\UNDOTBS01.DBF
destination for restore of datafile 00003: D:\ORACLE\PRODUCT\10.2.0\ORADATA\STDBY10G\SYSAUX01.DBF
destination for restore of datafile 00004: D:\ORACLE\PRODUCT\10.2.0\ORADATA\STDBY10G\USERS01.DBF
channel ORA_DISK_1: reading from backup piece D:\DB\DB_0OMD2B30_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=D:\DB\DB_0OMD2B30_1_1 tag=TAG20110523T144447
channel ORA_DISK_1: restore complete, elapsed time: 00:00:26
Finished recover at 23-MAY-11

RMAN>

after recovery, alter the standby database to recovery mode and check the archive status

SQL> recover managed standby database disconnect from session;
Media recovery complete.

SQL> select sequence#,archived,applied,deleted from v$archived_log where completion_Time like '23-MAY-11';

 SEQUENCE# ARC APP DEL
---------- --- --- ---
       175 YES NO  NO
       176 YES NO  NO
       177 YES NO  NO
       178 YES NO  NO
       179 YES NO  NO
       180 YES NO  NO
       181 YES NO  NO
       182 YES NO  NO
       183 YES NO  NO
       184 YES NO  NO
       188 YES YES NO

 SEQUENCE# ARC APP DEL
---------- --- --- ---
       189 YES YES NO
       185 YES YES NO
       186 YES YES NO
       187 YES YES NO
       190 YES YES NO
       191 YES YES NO
       192 YES YES NO
       193 YES YES NO
       194 YES YES NO
       195 YES YES NO
       196 YES YES NO

 SEQUENCE# ARC APP DEL
---------- --- --- ---
       197 YES YES NO
       198 YES YES NO
       199 YES YES NO

25 rows selected.

now check the SCN number in the standby.

SQL> select distinct checkpoint_change# from v$datafile_header order by 1;

CHECKPOINT_CHANGE#
------------------
            747276

If the archive file is corrupt or lost we dont need to rebuild the standby in 10g now.

Regards,
Navaneeth

Wednesday, May 11, 2011

Mapping a drive using command in windows

net use V: \\10.29.116.11\backup_dbs\Weekly username/user:password

User managed Cold backup & Hot backup mixup restoration scenario

How to restore a database which has a valid backup files taken from two different backup methods.

Suppose a database has 4 datafiles and On monday i have taken a backup of 2 datafiles using user managed cold backup (for some reasons didnt take backup of remaining 2 datafiles). And on tuesday i have taken the remaining 2 datafiles using user managed Hot backup.

And on friday i have lost the 3 datafiles of the database unfortunately. How to restore and recover using the backups taken using 2 different methods.


Please see the below steps

>> Monday taking backup of 1st 2 datafiles using User managed Cold backup.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>

SQL>host copy D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSTEM01.DBF D:\backup_10g\SYSTEM01.DBF
        1 file(s) copied.

SQL>host copy D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSAUX01.DBF D:\backup_10g\SYSAUX01.DBF
        1 file(s) copied.

>> On Monday evening there is some transactions happening.

SQL> conn scott/tiger
Connected.
SQL>
SQL> select * from tab;

TNAME                          TABTYPE  CLUSTERID
------------------------------ ------- ----------
DEPT                           TABLE
EMP                            TABLE
BONUS                          TABLE
SALGRADE                       TABLE

SQL> create table emp_1 as select * from emp;

Table created.

SQL>
SQL>
SQL> select * from tab;

TNAME                          TABTYPE  CLUSTERID
------------------------------ ------- ----------
DEPT                           TABLE
EMP                            TABLE
BONUS                          TABLE
SALGRADE                       TABLE
EMP_1                          TABLE

SQL> conn sys/database as sysdba
Connected.

SQL> alter system switch logfile;

System altered.

SQL> /

System altered.

SQL>
SQL> /

System altered.

SQL> /

System altered.

SQL> select name from v$archived_log;

     NAME
  --------- --- --- --- ------------ ---------------- ----------- --- --- --- ----------
  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_3_6WNDSTXM_.ARC

  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_2_6WNDSS09_.ARC

  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_4_6WNDT03Q_.ARC

  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_5_6WNDT5KD_.ARC

SQL>

>> On Tuesday we are taking the remaining 2 datafiles backup using user managed Hot backup method.

SQL> alter tablespace users begin backup;

Tablespace altered.

SQL> host copy D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\USERS01.DBF D:\backup_10g\USERS01.DBF
        1 file(s) copied.

SQL> alter tablespace users end backup;

Tablespace altered.

SQL> alter tablespace undotbs1 begin backup;

Tablespace altered.

SQL> host copy D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\UNDOTBS01.DBF d:\backup_10g\undotbs01.dbf
        1 file(s) copied.

SQL> alter tablespace undotbs1 end backup;

Tablespace altered.

>> Again there is some transaction happening on the weekdays until friday.


SQL> conn scott/tiger
Connected.
SQL> select * from tab;

TNAME                          TABTYPE  CLUSTERID
------------------------------ ------- ----------
DEPT                           TABLE
EMP                            TABLE
BONUS                          TABLE
SALGRADE                       TABLE
EMP_1                          TABLE

SQL> create table emp_2 as select * from emp;

Table created.

SQL> commit;

Commit complete.

SQL> conn sys/database as sysdba
Connected.
SQL> alter system switch logfile;

System altered.

SQL> /

System altered.

SQL> select * from v$archived_log;


 NAME
------- --- --- --- --- - --------- --- --- --- ------------ ---------------- ----------- --- --- ---
  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_3_6WNDSTXM_.ARC
  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_2_6WNDSS09_.ARC
  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_4_6WNDT03Q_.ARC
  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_5_6WNDT5KD_.ARC
  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_6_6WNFCP12_.ARC
  D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_7_6WNFCQ8W_.ARC


6 rows selected.

SQL> alter system switch logfile;

System altered.

SQL>

Assume the issue happens on Friday !!! the datafiles has been corrupted. So please see how it is restored in the below steps.

SQL> shutdown immediate
Database closed.
Database dismounted.

SQL>
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area  612368384 bytes
Fixed Size                  1250428 bytes
Variable Size             192940932 bytes
Database Buffers          411041792 bytes
Redo Buffers                7135232 bytes
Database mounted.
ORA-01157: cannot identify/lock data file 1 - see DBWR trace file
ORA-01110: data file 1: 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSTEM01.DBF'


SQL> select name from v$datafile;

NAME
------------------------------------------------------------
D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSTEM01.DBF
D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\UNDOTBS01.DBF
D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSAUX01.DBF
D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\USERS01.DBF

SQL> host copy D:\backup_10g\system01.dbf D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSTEM01.DBF
        1 file(s) copied.

SQL> host copy D:\backup_10g\sysaux01.dbf D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSAUX01.DBF
        1 file(s) copied.

SQL> host copy D:\backup_10g\USERS01.DBF D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\USERS01.DBF
        1 file(s) copied.

SQL> host copy D:\backup_10g\undotbs01.dbf D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\UNDOTBS01.DBF
        1 file(s) copied.

SQL> recover database;
ORA-00279: change 569282 generated at 05/11/2011 12:09:32 needed for thread 1
ORA-00289: suggestion : D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_2_%U_.ARC
ORA-00280: change 569282 for thread 1 is in sequence #2


Specify log: {=suggested | filename | AUTO | CANCEL}
AUTO
ORA-00279: change 569585 generated at 05/11/2011 12:26:56 needed for thread 1
ORA-00289: suggestion : D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_3_%U_.ARC
ORA-00280: change 569585 for thread 1 is in sequence #3
ORA-00278: log file 'D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_2_6WNDSS09_.ARC' no longer needed for this recovery


ORA-00279: change 569593 generated at 05/11/2011 12:26:58 needed for thread 1
ORA-00289: suggestion : D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_4_%U_.ARC
ORA-00280: change 569593 for thread 1 is in sequence #4
ORA-00278: log file 'D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_3_6WNDSTXM_.ARC' no longer needed for this recovery


ORA-00279: change 569596 generated at 05/11/2011 12:27:03 needed for thread 1
ORA-00289: suggestion : D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_5_%U_.ARC
ORA-00280: change 569596 for thread 1 is in sequence #5
ORA-00278: log file 'D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_4_6WNDT03Q_.ARC' no longer needed for this recovery


ORA-00279: change 569598 generated at 05/11/2011 12:27:09 needed for thread 1
ORA-00289: suggestion : D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_6_%U_.ARC
ORA-00280: change 569598 for thread 1 is in sequence #6
ORA-00278: log file 'D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL10G\ARCHIVELOG\2011_05_11\O1_MF_1_5_6WNDT5KD_.ARC' no longer needed for this recovery


Log applied.
Media recovery complete.
SQL> alter database open;

Database altered.

SQL> select status,name from v$datafile;

STATUS  NAME
------- --------------------------------------------------------------
SYSTEM  D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSTEM01.DBF
ONLINE  D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\UNDOTBS01.DBF
ONLINE  D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\SYSAUX01.DBF
ONLINE  D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL10G\USERS01.DBF

SQL>

Recovered completely without any issues.Check the below list of tables for verification.

SQL> conn scott/tiger
Connected.
SQL>
SQL> select * from tab;

TNAME                          TABTYPE  CLUSTERID
------------------------------ ------- ----------
DEPT                           TABLE
EMP                            TABLE
BONUS                          TABLE
SALGRADE                       TABLE
EMP_2                          TABLE
EMP_1                          TABLE

6 rows selected.

SQL>

Regards,
Navaneeth

Tuesday, May 3, 2011

10.2.0.2 Grid installation error in Windows 32 bit OS

While installing Oracle 10g R2 Grid software, the installation failed in the OMS configuration setup step.

Please see the error message as below



Check the installation logs for more information and try to find the exact error message.
In my installation i got the message below

"

opmnctl: starting opmn and all managed processes...
================================================================================
opmn id=COB413151:6200
    0 of 1 processes started.

ias-instance id=EnterpriseManager0.cob413151.cob.apac.bosch.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
    HTTP_Server/HTTP_Server/HTTP_Server

Error
--> Process (pid=4356)
    failed to start a managed process after the maximum retry limit
    Log:
    D:\Grid_home\oms10g\opmn\logs\HTTP_Server~1

Its a known bug in this version of the software, its mentioned in the below metalink docment.

"How to Recover From Enterprise Manager Grid Control 10g Base Releases (10.2.0.x.0) Installation Errors after 31-Dec-2010 [ID 1228103.1]As mentioned in the doc we have to install a patch to continue with the installation. Please follow the exact steps in the docs. it works fine after applying the patch.
---------------------------------------------------------------------------------------------------------------------------------


U:\>d:

D:\>cd Grid_home

D:\Grid_home>cd D:\10.2.0.2_grid_sftware

D:\10.2.0.2_grid_sftware>
D:\10.2.0.2_grid_sftware>cd 8430622

D:\10.2.0.2_grid_sftware\8430622>opatch
Oracle Home is not set. OPatch cannot proceed!

OPatch failed with error code = 1

D:\10.2.0.2_grid_sftware\8430622>set ORACLE_HOME=D:\Grid_home\oms10g\

D:\10.2.0.2_grid_sftware\8430622>opatch
Invoking OPatch 10.2.0.2.1

Oracle interim Patch Installer version 10.2.0.2.1
Copyright (c) 2005, Oracle Corporation.  All rights reserved..


Oracle Home       : D:\Grid_home\oms10g
Central Inventory : C:\Program Files\Oracle\Inventory
   from           : n/a
OPatch version    : 10.2.0.2.1
OUI version       : 10.2.0.2.0
OUI location      : D:\Grid_home\oms10g\\oui
Log file location : D:\Grid_home\oms10g\cfgtoollogs\opatch\opatch-2011_Apr_20_14-28-42-IST_Wed.log


 Usage: opatch [ -help ] [ -r[eport] ] [ command ]

            command := apply
                       lsinventory
                       query
                       rollback
                       version
                       prereq
                       util

  := -help       Displays the help message for the command.
                       -report     Print the actions without executing (deprecated).

 example:
   'opatch -help'
   'opatch apply -help'
   'opatch lsinventory -help'
   'opatch rollback -help'
   'opatch prereq -help'
   'opatch util -help'



OPatch succeeded.

D:\10.2.0.2_grid_sftware\8430622>opatch apply
Invoking OPatch 10.2.0.2.1

Oracle interim Patch Installer version 10.2.0.2.1
Copyright (c) 2005, Oracle Corporation.  All rights reserved..


Oracle Home       : D:\Grid_home\oms10g
Central Inventory : C:\Program Files\Oracle\Inventory
   from           : n/a
OPatch version    : 10.2.0.2.1
OUI version       : 10.2.0.2.0
OUI location      : D:\Grid_home\oms10g\\oui
Log file location : D:\Grid_home\oms10g\cfgtoollogs\opatch\opatch-2011_Apr_20_14-28-48-IST_Wed.log

ApplySession applying interim patch '8430622' to OH 'D:\Grid_home\oms10g'

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '8430622' for restore. This might take a while...
Backing up files affected by the patch '8430622' for rollback. This might take a while...

Patching component oracle.sysman.top.oms, 10.2.0.2.0...
Updating jar file "D:\Grid_home\oms10g\sysman\jlib\emagentSDK.jar" with "\sysman\jlib\emagentSDK.jar\oracle\sysman\eml\sec\util\RootCert.class"
ApplySession adding interim patch '8430622' to inventory

Verifying the update...
Inventory check OK: Patch ID 8430622 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 8430622 are present in Oracle Home.

OPatch succeeded.

D:\10.2.0.2_grid_sftware\8430622>cd ..

D:\10.2.0.2_grid_sftware>cd ..

D:\>cd Grid_home

D:\Grid_home>cd oms10g

D:\Grid_home\oms10g>cd bin

D:\Grid_home\oms10g\BIN>emctl start oms
Oracle Enterprise Manager 10g Release 10.2.0.2.0
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
opmnctl: opmn started
Starting HTTP Server ...
Starting Oracle Management Server ...
Checking Oracle Management Server Status ...
Oracle Management Server is not functioning because of the following reason:
Unexpected error occurred. Check error and log files.

D:\Grid_home\oms10g\BIN>emctl start oms
Oracle Enterprise Manager 10g Release 10.2.0.2.0
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
opmnctl: opmn already running
Starting HTTP Server ...
Starting Oracle Management Server ...
Checking Oracle Management Server Status ...
Oracle Management Server is not functioning because of the following reason:
Unexpected error occurred. Check error and log files.

D:\Grid_home\oms10g\BIN>emctl status oms
Oracle Enterprise Manager 10g Release 10.2.0.2.0
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
Oracle Management Server is not functioning because of the following reason:
Unexpected error occurred. Check error and log files.

D:\Grid_home\oms10g\BIN>emctl
Oracle Enterprise Manager 10g Release 10.2.0.2.0
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
   Oracle Enterprise Manager 10g Grid Control commands:
       emctl start| stop| status| setpasswd| getversion oms
       emctl config oms sso -host ssoHost -port ssoPort -sid ssoSid -pass ssoPassword -das dasURL -u user
       emctl config oms loader -shared -dir
       emctl config| status emkey

       emctl start | stop iasconsole
       emctl status iasconsole
       emctl set password
       emctl secure iasconsole
       emctl unsecure iasconsole/em
       emctl status agent
       emctl status agent -secure [-omsurl :/em/*>]
       emctl getversion
       emctl reload | upload | clearstate | getversion agent
       emctl reload agent dynamicproperties [:]....
       emctl config agent
       emctl config agent updateTZ
       emctl config agent getTZ
       emctl resetTZ agent
       emctl config agent credentials [[:]]
 Blackout Usage :
       emctl start blackout [-nodeLevel] [[:]].... [-d ]
       emctl stop blackout
       emctl status blackout [[:]]....

The following are valid options for blackouts
defaults to local node target if not specified.
If -nodeLevel is specified after ,the blackout will be applied to all targets and any target list that follows will be ignored.
 Duration is specified in [days] hh:mm

        emctl getemhome
        emctl ilint
        emctl start | stop | status subagent
 Register Targettype Usage :
emctl register oms targettype [-o ]