Tuesday, October 21, 2014

Oracle ASR snmp notification solution for Database Compute Nodes

Oracle ASR - Automatic Service Request is a secure automatic service request generation and priority service request handling for hardware faults on Oracle engineered systems such as Exadata and the ZFS storage appliance. When a hardware fault is detected, Oracle Auto Service Request opens a service request with Oracle automatically and transports electronic fault telemetry to help 
expedite the diagnostic process. Oracle Support is notified promptly and parts are dispatched upon receipt of the service request if required.

I was working on setting up ASR on a couple of Exadata systems for a client and noticed the asrexachk script did not get any snmp notification from the Database compute nodes. The asrexachk script is designed to check and test ASR configurations to make sure that the Engineered Systems can communicate to the ASR Manager server.

Oracle Support suggested that the correct route on the Exadata compute nodes may not be taken by snmp to send a message via the UDP protocol on port 162 and that a manual static route may need to be added to the Exadata database compute nodes,

The following commands were given as a solution to run on all database compute nodes.

First you need to get the gateway IP address from the server.



$ grep -i gateway /etc/sysconfig/network-scripts/ifcfg-eth0
GATEWAY=<GATEWAY IP Address>

Then you can add the static route.

$ /sbin/route add -host <ASR Manager Server IP Address> gw <GATEWAY IP Address>


Run the tcpdump command from the ASR Manager server to see what notifications are sent in. You will see after the above route changes snmp trap notifications will start to come in from the DB nodes when an test notification is generated from the DB compute nodes.

[root@ASRMANAGER ~]# tcpdump -ni eth1 udp port 162
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
22:06:28.712136 IP <DB node IP address>.33020 > <ASR Manager Server IP Address>.snmptrap:  Trap(24)  .1.3.6.1.6.3 10.43.47.133 coldStart 0   <--- SNMP TRAP DB NODE
22:08:36.931920 IP <DB node IP address>.12984 > <ASR Manager Server IP Address>.snmptrap:  V2Trap(286)  .1.3.6.1.2.1.1.3.0=206723517 [|snmp] <--- DB NODE ASR SNMP TRAP




From the Database compute nodes you can run the snmptrap command to send to the ASR Manager server and you will see the above message.


[root@EXANODE1 ~]# snmptrap -v 1 -c public <ASR Manager Server IP Address> .1.3.6.1.6.3 "" 0 0 coldStart.0

From the Database compute nodes you can run the exadata_mon_hw_asr.pl command to send to the ASR Manager server and you will see the above message.


[root@EXANODE1 ~]# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -validate_snmp_subscriber -type asr

Sending test trap from <DB NODE 1 IP ADDRESS> to destination - <ASR Manager Server IP Address>:162

Then an email is sent as follows and it comes from no.reply@oracle.com and is with the subject - Oracle ASR: *Test* Service Request.


Serial#: XXXXXXXXX
Hostname: <DB NODE>
Service Request test-create was successful.
The Oracle Auto Service Request documentation can be accessed on http://oracle.com/asr.
Please use My Oracle Support https://support.oracle.com for assistance. 

The below table is a reference guide for the ports requirements for the ASR Manager server and also the source, destination, protocol and description information.

Monday, October 13, 2014

Enterprise Manager 12c No Special Characters for SYSMAN account

While doing an upgrade of Oracle Enterprise Manager 12c from 12.1.0.1 to 12.1.0.4 I encountered an error which caused the upgrade to fail. Please keep in mind you cannot do a direct path upgrade from EM 12.1.0.1 to 12.1.0.4 you will first need to apply Bundle Patch (BP1) to the 12.1.0.1 binaries and then you can upgrade from 12.1.0.1 BP1 --> 12.1.0.2 or 12.1.0.3 and then finally you can upgrade to 12.1.0.4.

From the upgrade logs the error stack is below.


INFO: oracle.sysman.top.oms:MDS Schema Creation status: FAILURE 
SEVERE: oracle.sysman.top.oms:MDS Schema Creation is failed 
INFO: oracle.sysman.top.oms:_______________________________________________________________________ 
INFO: oracle.sysman.top.oms:Starting opss Schema Creation. 
INFO: oracle.sysman.top.oms:Failed to create OPSS schema 
INFO: oracle.sysman.top.oms:_______________________________________________________________________ 
INFO: oracle.sysman.top.oms:Starting APM Schema Creation. 
INFO: oracle.sysman.top.oms:APM Component is already created 
INFO: oracle.sysman.top.oms:The plug-in MDS Schema Configuration has failed its perform method 

Cannot change password for sysman_mds java.sql.SQLSyntaxErrorException: ORA-00922: missing or invalid option 

ERROR:MDS java.sql.SQLSyntaxErrorException: ORA-00922: missing or invalid option 

java.sql.SQLSyntaxErrorException: ORA-00922: missing or invalid option 



The Upgrade basically failed due to special character(s) in the SYSMAN password. Please make sure you do not have any special characters in the SYSMAN password such as *,#,!, etc Just use a good combination of alpha and number characters.

Since the upgrade failed I had to fallback and do the following to restart the upgrade all over again for this issue.

1.) Restore the database, inventory and OMS Middleware Home from backup.
2.) Startup database and listener
3.) Start the OMS
4.) Change the SYSMAN password that doesn't use special character(s), procedure below.


<OMS_HOME>/bin/emctl config oms -change_repos_pwd

Please note changing the SYSMAN password will change it on both the OMS and on the repository database.

Enterprise Manager 12c Patching changes

Starting with Enterprise Manager 12.1.0.4.3 (PS3) all OMS bundle patches and plugins will be released as a single OMS-side "System" patch.  The patch will be installed using the OPatchauto installation method.  The EM 12.1.0.4.0 Agent and Agent-side Plug-in Bundle patches will continue to use Opatch.

More details at MOS: Enterprise Manager 12.1.0.4.0 (PS3) Master Bundle Patch List for Agent and Plug-ins (OMS and Agent side) (Doc ID 1900943.1)

Wednesday, October 1, 2014

Bash security alert on Exadata/Oracle Linux

There is a new security issue named "Bash Shell Shock" also documented as CVE-2014-6271/CVE-2014-7169.

Shellshock is the nickname for a flaw in Bourne Shell, or Bash, which is commonly used on Oracle Enterprise Linux (OEL). The Bash Shell Shock vulnerability allows remote attackers to write to files or possibly have unknown other impact.

On database nodes, obtain and update updated bash package using the following version, or later:
bash-3.2-33.el5_11.4.x86_64

This package may be obtained via public-yum.oracle.com.

To install this package on DB nodes, the exadata-sun-computenode-exact RPM must be removed first. If using Exadata DB server image version 11.2.3.3.0 or later, first run this command: rpm -e exadata-sun-computenode-exact

Then, use this command on all releases to install the updated rpm.
rpm -Uvh <new bash rpm>

To install this package on storage cells (supported as an exception for this CVE only), install using "rpm -Uvh --nodeps <path to bash rpm>"

Stay up to date by regularly check My Oracle Support(MOS) Note for the latest security finding.

Responses to common Exadata security scan findings (Doc ID 1405320.1)


The following command can be used to see if you are affected by the issue, if the command returns vulnerable then you are at risk.

[root@localhost ~]# env 'x=() { :;}; echo vulnerable' 'BASH_FUNC_x()=() { :;}; e                                                     cho vulnerable' bash -c "echo this is a test"
vulnerable
bash: BASH_FUNC_x(): line 0: syntax error near unexpected token `)'
bash: BASH_FUNC_x(): line 0: `BASH_FUNC_x() () { :;}; echo vulnerable'
bash: error importing function definition for `BASH_FUNC_x'
this is a test


Below I have included the before and after results of removing the Shell shock issue from a OEL server.

[root@localhost ~]# yum update bash
Loaded plugins: downloadonly, security
public_ol6_UEKR3_latest                                                                                       | 1.2 kB     00:00
public_ol6_latest                                                                                             | 1.4 kB     00:00
public_ol6_latest/primary                                                                                     |  41 MB     00:37
public_ol6_latest                                                                                                        26205/26205
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package bash.x86_64 0:4.1.2-15.el6_4 will be updated
---> Package bash.x86_64 0:4.1.2-15.el6_5.2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================
 Package                 Arch                      Version                                Repository                            Size
=====================================================================================================================================
Updating:
 bash                    x86_64                    4.1.2-15.el6_5.2                       public_ol6_latest                    905 k

Transaction Summary
=====================================================================================================================================
Upgrade       1 Package(s)

Total download size: 905 k
Is this ok [y/N]: y
Downloading Packages:
bash-4.1.2-15.el6_5.2.x86_64.rpm                                                                              | 905 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : bash-4.1.2-15.el6_5.2.x86_64                                                                                      1/2
  Cleanup    : bash-4.1.2-15.el6_4.x86_64                                                                                        2/2
  Verifying  : bash-4.1.2-15.el6_5.2.x86_64                                                                                      1/2
  Verifying  : bash-4.1.2-15.el6_4.x86_64                                                                                        2/2

Updated:
  bash.x86_64 0:4.1.2-15.el6_5.2

Complete!
[root@localhost ~]# env 'x=() { :;}; echo vulnerable' 'BASH_FUNC_x()=() { :;}; echo vulnerable' bash -c "echo this is a test"
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `BASH_FUNC_x'
this is a test
[root@localhost ~]#





Wednesday, August 20, 2014

Speaking tomorrow at the St. Louis Oracle Users Group (SLOUG) about Database as a Service (DBaaS)

I will be speaking tomorrow at the St. Louis Oracle Users Group (SLOUG) about Database as a Service (DBaaS) . Easily with just a click a user can request a database. Learn how to deploy a Single and RAC 12c Database within minutes from OEM 12c from start to finish.
Now you can automate from the cloud OEM12c creating a clone of a production database delivered with Data Masking and Security/Encryption with TDE!

Aug 21, 2014 @1PM Central Time.


Oracle Office St. Louis
424 South Woods Mill Road, suite 200
Chesterfield, MO 63017

Friday, August 1, 2014

Recreate Oracle Home Inventory

Trying to upgrade an Oracle Home and discovered the Oracle Home was corrupted and had to rebuild it.

The steps are very simple.

I just followed the note: Steps To Recreate Central Inventory(oraInventory) In RDBMS Homes (Doc ID 556834.1)


1. I ran the OPatch command and it failed, highlighted in Green below.

[oracle@server1] $  $ORACLE_HOME/OPatch/opatch lsinventory -all
Invoking OPatch 11.1.0.6.2

Oracle Interim Patch Installer version 11.1.0.6.2
Copyright (c) 2007, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/oracle/product/11.1.0
Central Inventory : /u01/oracle/product/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.2
OUI version       : 11.1.0.7.0
OUI location      : /u01/oracle/product/11.1.0//oui
Log file location : /u01/oracle/product/11.1.0/cfgtoollogs/opatch/opatch2014-08-02_01-31-56AM.log

List of Homes on this system:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo

OPatch failed with error code 73


2. Rename the existing corrupted oraInventory directory listed in the /etc/oraInst.loc file.

[oracle@server1] $ ls -ld /u01/oracle/product/oraInventory
drwxr-sr-x    3 oracle   dba             256 Jul 30 18:20 /u01/oracle/product/oraInventory
[oracle@server1] $ cd /u01/oracle/product/

[oracle@server1] $ mv oraInventory oraInventory_old_Aug_2_2014


3. Go to the $ORACLE_HOME/oui/bin directory

[oracle@server1] $ cd $ORACLE_HOME/oui/bin
[oracle@server1] $ pwd
/u01/oracle/product/11.1.0/oui/bin
[oracle@server1] $ echo $ORACLE_HOME
/u01/oracle/product/11.1.0/


4. Attach the Oracle Home and recreate the inventory with the runInstaller command.

[oracle@server1] $ ./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="/u01/oracle/product/11.1.0" ORACLE_HOME_NAME="11gR1"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16704 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-08-02_01-36-41AM. Please wait ...[oracle@server1] $ The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/oracle/product/oraInventory

[oracle@server1] $ 'AttachHome' was successful.


5. Now the Oracle Home inventory is fine, whala! The opatch command will run fine now.

[oracle@server1] $ $ORACLE_HOME/OPatch/opatch lsinventory -all
Invoking OPatch 11.1.0.6.2

Oracle Interim Patch Installer version 11.1.0.6.2
Copyright (c) 2007, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/oracle/product/11.1.0
Central Inventory : /u01/oracle/product/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.2
OUI version       : 11.1.0.7.0
OUI location      : /u01/oracle/product/11.1.0//oui
Log file location : /u01/oracle/product/11.1.0/cfgtoollogs/opatch/opatch2014-08-02_01-38-46AM.log

Lsinventory Output file location : /u01/oracle/product/11.1.0/cfgtoollogs/opatch/lsinv/lsinventory2014-08-02_01-38-46AM.txt

--------------------------------------------------------------------------------
List of Oracle Homes:
  Name          Location
   11gR1         /u01/oracle/product/11.1.0

Installed Top-level Products (2):

Oracle Database 11g                                                  11.1.0.6.0
Oracle Database 11g Patch Set 1                                      11.1.0.7.0
There are 2 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


--------------------------------------------------------------------------------

OPatch succeeded.

Monday, July 28, 2014

Always run the pre-upgrade script before a database upgrade!

I was in a hurry to upgrade a database from 11.1.0.7 to 11.2.0.4 and encountered the following error message: ORA-1722 when running the catupgrd.sql script. The script exited with the error message I just listed.

The root cause of the problem is that you must always run the utlu112i from the $ORACLE_HOME/rdbms directory before you upgrade your database. This script is not optional, it is actually mandatory since it makes changes to the database dictionary for the upgrade preparation.

I also suggest to please create a flashback guaranteed restore point just before kicking off the upgrade just in case you need to flash your database back. This can save valuable hours to flashback a database rather than restoring it from a backup. You can simply flashback the database to the GRP - guaranteed restore point if required rather than doing a full database restore which may take several hours or more from a disk based backup.




Wednesday, July 2, 2014

Speaking at Dallas Oracle Users Group(DOUG) on July 24, 2014

I will be speaking at the Dallas Oracle Users Group - DOUG on July 24, 2014. Please come and learn about Exadata and ZFS upgrades!

http://tinyurl.com/naa46l8

Here is the information about the presentation below.

Exadata is one of Oracle's flagship engineered systemsand the ZFS Storage appliance is a premier storage system and keeping them both healthy and up to date can be easily achieved by upgrading and patching. Ultimately you can save on thousands of dollars on overall support costs, avoid unplanned downtime, software bugs and also take advantage of new features by ensuring you have a regular schedule to patch your Exadata and ZFS systems. It is a common impression that patching the Exadata and ZFS platform is complicated and thorough however you can learn about the process and we can help you to understand it and upgrade with minimal downtime with quick efficient methods. We will discuss strategies to upgrading the Exadata full stack and ZFS to mitigate risk and reduce downtime, we will show you how to avoid mistakes and best practices to implement and plan your upgrades. We will also share a detailed case study of one customer for a large fortune 500 healthcare supply chain company who did not upgrade their Exadata and ZFS for almost 3 years and how we upgraded it flawlessly with zero issues post-upgrade

Wednesday, April 16, 2014

Upgrade your Database to 12c Superfast!

Most DBAs know about the catupgrd.sql script Upgrade Utility to process the upgrade singled threaded. Now there is a faster way to upgrade your Oracle Database to 12c, you can use the catctl.pl the Parallel Upgrade Utility – a perl script, which replaces catupgrd.sql and provides both parallel processing mode and serial mode. This will greatly help reduce downtime for a database upgrade to 12c and I highly recommend you use this option.

The –n option Specifies the number of processes to use for parallel operations (default = 4). The maximum is 8. Set this parameter to 0 to run catctl.pl in serial mode.

The catctl.pl Parallel Upgrade Utility is integrated with DBUA the gui upgrade utility. However, for manual, command line upgrades, you can run catctl.pl with various parameters

If -n 3 is used when invoking catctl.pl, then catupgrd0.log, catupgrd1.log, catupgrd2.log are created during the upgrade process. When confirming if an upgrade ran fine or failed, all of the catupgrd#.log files must be checked. If catctl.pl is rerun, then the previous log files are overwritten unless you specify a different log directory.

To run catctl.pl on Linux from the new Oracle 12c home.


cd $ORACLE_HOME/rdbms/admin
$ORACLE_HOME/perl/bin/perl catctl.pl -n 3 -l $ORACLE_HOME/diagnostics catupgrd.sql


Once you start the database in upgrade mode by issuing the following command:
SQL> STARTUP UPGRADE

The UPGRADE keyword prepares the database for the upgrade.

Once the database is started in upgrade mode, only queries on fixed views execute without errors until after the catctl.pl script is run. Before running catctl.pl, queries on any other view or the use of PL/SQL returns an error.

If errors appear listing desupported initialization parameters, then make a note of the desupported initialization parameters and continue with the upgrade. Remove the desupported initialization parameters the next time you shut down the database.

Steps to Rename an Oracle Enterprise Manager Repository Database

Steps to rename an OEM repository Database.

This is something you may not have to do very often. I will show you how to rename your Oracle Enterprise Manager 12c repository database.

Take a full RMAN backup with archivelogs.

Shutdown OEM

$ . oraenv
ORACLE_SID = [oem] ? oem
The Oracle base for ORACLE_HOME=/opt/apps/MW/oms is /home/oracle
[oracle@dnvappoem06 ~]$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...

Change the cluster database parameter to false

SQL> alter system set cluster_database=false scope=spfile sid='*';
System altered.

Backup the spfile
SQL> create pfile='/tmp/initent_oem.ora' from spfile;

File created.

Update the init file /tmp/initent_oem.ora with new parameters for the new sid and db name
Stop the database

/home/oracle>srvctl stop database -d oemsblp

In ASM make the new directories
mkdir +DATA/ent_oem/tempfile
mkdir +DATA/ent_oem/datafile


Startup mount only on one of the instances 

Invoke the utility on the command line, specifying a valid user with the SYSDBA privilege.  You must specify both the DBNAME and SETNAME parameters. The SETNAME parameter tells the DBNEWID utility to only alter the database name. 
  
% /home/oracle>nid TARGET=/  DBNAME=ent_oem SETNAME=YES

DBNEWID: Release 11.2.0.3.0 - Production on Thu Feb 13 20:44:01 2014

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Connected to database OEMSBLP (DBID=1189579321)

Connected to server version 11.2.0

Control Files in database:
    +RECO/oemsblp/controlfile/current.261.832085049

Change database name of database OEMSBLP to ENT_OEM? (Y/[N]) => Y

Proceeding with operation
Changing database name from OEMSBLP to ENT_OEM
    Control File +RECO/oemsblp/controlfile/current.261.832085049 - modified
    Datafile +DATA/oemsblp/datafile/system.293.83208499 - wrote new name
    Datafile +DATA/oemsblp/datafile/sysaux.275.83208499 - wrote new name
    Datafile +DATA/oemsblp/datafile/undotbs1.276.83208499 - wrote new name
    Datafile +DATA/oemsblp/datafile/users.290.83208499 - wrote new name
    Datafile +DATA/oemsblp/datafile/undotbs2.292.83208513 - wrote new name
    Datafile +DATA/oemsblp/datafile/mgmt_ecm_depot_ts.287.83232853 - wrote new name
    Datafile +DATA/oemsblp/datafile/mgmt_tablespace.270.83232853 - wrote new name
    Datafile +DATA/oemsblp/datafile/mgmt_ad4j_ts.264.83232853 - wrote new name
    Datafile +DATA/oemsblp/tempfile/temp.291.83208507 - wrote new name
    Control File +RECO/oemsblp/controlfile/current.261.832085049 - wrote new name
    Instance shut down

Database name changed to ENT_OEM.
Modify parameter file and generate a new password file before restarting.
Succesfully changed database name.
DBNEWID - Completed succesfully.

/home/oracle>





Create a new init file pointer in the $ORACLE_HOME/dbs on each node

NODE1:
/opt/oracle/product/11203/db_1/dbs>cat initent_oem1.ora
SPFILE='+DATA/ent_oem/spfileent_oem.ora'

NODE2:
/opt/oracle/product/11203/db_1/dbs>cat initent_oem2.ora
SPFILE='+DATA/ent_oem/spfileent_oem.ora'

 Create a new password file on each node from the $ORACLE_HOME/dbs directory.

NODE1:
cp orapwoemsblp1 orapwent_oem1

NODE2:
cp orapwoemsblp1 orapwent_oem2

create spfile='+DATA/ent_oem/spfileent_oem.ora' from pfile='/tmp/initent_oem.ora' ;

Set the new ORACLE_SID
Verify the parameters are updated correctly to the new name, note the instance_name and db_name and other parameters.
SQL> show parameter name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_file_name_convert                 string
db_name                              string      ent_oem
db_unique_name                       string      ent_oem
global_names                         boolean     FALSE
instance_name                        string      ent_oem1
lock_name_space                      string
log_file_name_convert                string
processor_group_name                 string
service_names                        string      ent_oem
SQL> show parameter pfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/ent_oem/spfileent_oem.ora
                                                 


Change the cluster_database parameter back to true
SQL> alter system set cluster_database=true scope=spfile sid='*';

DBNEWID utility makes the changes in the controlfile and datafile headers. This utility is not RAC aware. So it will not update the OCR when the database is renamed. Hence, the user needs to invoke SRVCTL to remove and again add the database information to OCR:

Before the Rename
srvctl stop database -d
srvctl remove instance -d oemsblp -i oemsblp1
srvctl remove instance -d oemsblp -i oemsblp2
 (repeat above command for each instance as needed)
srvctl remove database -d oemsblp

And after the Rename
srvctl add database -d ent_oem -o /opt/oracle/product/11203/db_1
/opt/oracle/product/11203/db_1/dbs>srvctl add instance -d ent_oem -i ent_oem1 -n dnvdboem01
/opt/oracle/product/11203/db_1/dbs>srvctl add instance -d ent_oem -i ent_oem2 -n dnvdboem02

(repeat above command for each instance as needed)

Start the database
srvctl start database –d ent_oem


Save a backup copy of the controlfile of the database.
alter database backup controlfile to trace as '/tmp/ent_oem.trc';

Change the cluster_mode back to false

alter system set cluster_database=false scope=spfile sid='*';




Shutdown the database and restart in mount mode.


Copy and rename the datafiles to the new directory location via RMAN while connected to the $ORACLE_SID
copy datafile '+DATA/oemsblp/datafile/system.293.832084993' to '+data';
copy datafile '+DATA/oemsblp/datafile/sysaux.275.832084993' to '+data';
copy datafile '+DATA/oemsblp/datafile/undotbs1.276.832084993' to '+data';
copy datafile '+DATA/oemsblp/datafile/users.290.832084993' to '+data';
copy datafile '+DATA/oemsblp/datafile/undotbs2.292.832085133' to '+data';
copy datafile '+DATA/oemsblp/datafile/mgmt_ecm_depot_ts.287.832328533' to '+data';
copy datafile '+DATA/oemsblp/datafile/mgmt_tablespace.270.832328533' to '+data';
copy datafile '+DATA/oemsblp/datafile/mgmt_ad4j_ts.264.832328533' to '+data';



ALTER TABLESPACE TEMP ADD TEMPFILE '+DATA' size 1000M;

Run the following controlfile script with the new directory locations

STARTUP NOMOUNT
CREATE CONTROLFILE reuse DATABASE "ENT_OEM" NORESETLOGS  ARCHIVELOG
    MAXLOGFILES 192
    MAXLOGMEMBERS 3
    MAXDATAFILES 1024
    MAXINSTANCES 32
    MAXLOGHISTORY 584
LOGFILE
  GROUP 1 '+RECO/ent_oem/onlinelog/group_1.262.832085053'  SIZE 500M BLOCKSIZE 512,
  GROUP 2 '+RECO/ent_oem/onlinelog/group_2.263.832085057'  SIZE 500M BLOCKSIZE 512,
  GROUP 3 '+RECO/ent_oem/onlinelog/group_3.265.832085239'  SIZE 500M BLOCKSIZE 512,
  GROUP 4 '+RECO/ent_oem/onlinelog/group_4.266.832085247'  SIZE 500M BLOCKSIZE 512,
  GROUP 5 '+RECO/ent_oem/onlinelog/group_5.264.832085061'  SIZE 500M BLOCKSIZE 512,
  GROUP 6 '+RECO/ent_oem/onlinelog/group_6.267.832085253'  SIZE 500M BLOCKSIZE 512
-- STANDBY LOGFILE
DATAFILE
  '+DATA/ent_oem/datafile/system.293.832084993',
  '+DATA/ent_oem/datafile/sysaux.275.832084993',
  '+DATA/ent_oem/datafile/undotbs1.276.832084993',
  '+DATA/ent_oem/datafile/users.290.832084993',
  '+DATA/ent_oem/datafile/undotbs2.292.832085133',
  '+DATA/ent_oem/datafile/mgmt_ecm_depot_ts.287.832328533',
  '+DATA/ent_oem/datafile/mgmt_tablespace.270.832328533',
  '+DATA/ent_oem/datafile/mgmt_ad4j_ts.264.832328533'
CHARACTER SET AL32UTF8
;

ALTER TABLESPACE TEMP ADD TEMPFILE '+DATA' size 1000M;

On the OMS server run the following command to change the OMS repository database name

[oracle@dnvappoem06 ~]$ emctl config oms -store_repos_details -repos_conndesc '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)
(HOST=doem1dbrac-scan)(PORT=1521)))(LOAD_BALANCE=ON)
> (CONNECT_DATA=(SERVICE_NAME=ent_oem)))' -repos_user sysman
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Repository User's Password :

Successfully updated datasources and stored repository details in Credential Store.
If there are multiple OMSs in this environment, run this store_repos_details command on all of them.
And finally, restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
[oracle@dnvappoem06 log]$ emctl config oms -list_repos_details
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Repository Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=doem1dbrac-scan)(PORT=1521)))(LOAD_BALANCE=ON)
(CONNECT_DATA=(SERVICE_NAME=ent_oem)))
Repository User : sysman

[oracle@dnvappoem06 ~]$
[oracle@dnvappoem06 ~]$ emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down

[oracle@dnvappoem06 ~]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

[oracle@dnvappoem06 ~]$ emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
WebTier is Up

Oracle Management Server is Up