Tuesday, December 28, 2010

Good reference for RAC/Grid

http://psoug.org/reference/rac.html

Step By Step process of Configuring Oracle 10gR2 (10.2.0.3) RAC Standby For RAC Primary with ASM and OMFs. http://www.oracledba.org/10g/dr/10gR2_dataguard_RAC_to_RAC.html#CREATE_PHYSICAL_STANDBY_Database:

Oracle® Real Application Clusters Administrator's Guide
http://www.stanford.edu/dept/itss/docs/oracle/10g/rac.101/b10765/toc.htm

Thursday, October 28, 2010

DBCA failed at the end on 11gR2 grid

11.2 grid and 10g Oracle home. DBCA can create the database all the way to 100%. But after creating the database, it failed to start.

Solution:

Looking at Oracle Doc


Incorrect Ownership and Permission after Relinking or Patching 11gR2 Grid Infrastructure [ID 1083982.1]

Thursday, October 14, 2010

Error when try to connect 11g ASM using 10g DBCA

11g R2 RAC environemnt and successfully installed 10.2.0.4. While runing dbca to create a database, got the following error

"DBCA could not startup the ASM instance configured on this node. To processd with database creation using ASM you need the ASM instance to be up and running. Do you want to recreate the ASM instance on this node?"

Solution, according to Oracle Note

Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment [ID 948456.1], applied patch 8288940.

Saturday, October 9, 2010

ORA-29701: unable to connect to Cluster Synchronization Service

11g R2 grid infrustructure on a standalone server. After the Server moved from one data center to another, go the following errors while start the ASM.

SQL> startup
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service

Solution:

run crsctl start resource -all.

[oracle@passl07 bin]$ ./crsctl start resource -all
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'passl07'
CRS-2672: Attempting to start 'ora.cssd' on 'passl07'
CRS-2679: Attempting to clean 'ora.diskmon' on 'passl07'
CRS-2681: Clean of 'ora.diskmon' on 'passl07' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on 'passl07'
CRS-2676: Start of 'ora.diskmon' on 'passl07' succeeded
CRS-2676: Start of 'ora.cssd' on 'passl07' succeeded
CRS-5014: Agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" for action "start": details at "(:CLSN00009:)" in "/u01/app/oracle/product/11.2.0/grid/log/passl07/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
CRS-5014: Agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00009:)" in "/u01/app/oracle/product/11.2.0/grid/log/passl07/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
(:CLSN00009:)Utils:execCmd aborted
CRS-2674: Start of 'ora.LISTENER.lsnr' on 'passl07' failed
CRS-2679: Attempting to clean 'ora.LISTENER.lsnr' on 'passl07'
CRS-2678: 'ora.LISTENER.lsnr' on 'passl07' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-2672: Attempting to start 'ora.asm' on 'passl07'
CRS-2676: Start of 'ora.asm' on 'passl07' succeeded
CRS-2672: Attempting to start 'ora.PASS_DATA.dg' on 'passl07'
CRS-2672: Attempting to start 'ora.PASS_FLASH.dg' on 'passl07'
CRS-2672: Attempting to start 'ora.PASS_LOG.dg' on 'passl07'
CRS-2672: Attempting to start 'ora.PASS_BACKUP.dg' on 'passl07'
CRS-2676: Start of 'ora.PASS_FLASH.dg' on 'passl07' succeeded
CRS-2676: Start of 'ora.PASS_BACKUP.dg' on 'passl07' succeeded
CRS-2676: Start of 'ora.PASS_DATA.dg' on 'passl07' succeeded
CRS-2676: Start of 'ora.PASS_LOG.dg' on 'passl07' succeeded

Tuesday, September 14, 2010

ld: warning: libclntsh.so.9.0, needed by /psoft/psoftdev/hcmdev/bin/libpsora_ansi.so, not found (try using -rpath or -rpath-link)

New installation of PeopleSoft People Tools 8.49. Cobol compiled fine but got the following errors when linking:

[psoft@hcmdevdb01 setup]$ ./psrun.mak
./psrun.mak - linking PSRUN for redhat-4-ia32, Version 2.6.18-164.el5 ...
./psrun.mak - Error(s) encountered creating PSRUN!
./psrun.mak - See /psoft/psoftdev/hcmdev/setup/psrun.err for messages
[psoft@hcmdevdb01 setup]$ cat psrun.err
ld: warning: libclntsh.so.9.0, needed by /psoft/psoftdev/hcmdev/bin/libpsora_ansi.so, not found (try using -rpath or -rpath-link)
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIStmtExecute'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIStmtPrepare'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIInitialize'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCITransCommit'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIBindByName'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIAttrSet'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIDescriptorAlloc'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIStmtFetch'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIDescriptorFree'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIDefineByPos'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIHandleFree'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIStmtSetPieceInfo'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCISessionEnd'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIServerAttach'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIEnvInit'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIDateTimeGetDate'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIHandleAlloc'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIAttrGet'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIDateTimeGetTime'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIStmtGetPieceInfo'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIErrorGet'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCITransRollback'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCISessionBegin'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIServerVersion'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIServerDetach'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIDateTimeConstruct'
/psoft/psoftdev/hcmdev/bin/libpsora_ansi.so: undefined reference to `OCIParamGet

By searching in Google and Oracle support, all suggested to create a soft link. So I went to create the soft link under $ORACLE_HOME/lib as

ln -s /u01/app/oracle/product/10.2.0/db_1/lib32/libclntsh.so.10.1 libclntsh.so.9.0

But still it didn't work.

Finally, I figured out the problem is the permission. After applied the Oracle patch set, the permission on the Oracle home is tight. So I did the following:

cd $ORACLE_HOME/install
[oracle@hcmdevdb01 install]$ sh changePerm.sh

-------------------------------------------------------------------------------
Disclaimer: The purpose of this script is to relax permissions on some of the
files in the database Oracle Home so that all clients can access them.
Please note that Oracle Corporation recommends using the most restrictive file
permissions as possible for your given implementation. Running this script
should be done only after considering all security ramifications.
-------------------------------------------------------------------------------

Do you wish to continue (y/n) [n]: y
Spooling the error log /tmp/changePerm_err.log...
Finished running the script successfully
[oracle@hcmdevdb01 install]$

Then the Cobol link runs successfully.

Friday, September 10, 2010

Cloning an Oracle Home

I accidently installed Oracle on the wrong directory. So I decide to clone the home installed to the new location and delete the original one. Here is what I did:

1. Copy the home to the new location:

cp -rp product /u01/app/oracle

2. Go to the new home location, cd oui/bin and then run
./runInstaller -clone ORACLE_BASE="/u01/app/oracle" ORACLE_HOME="/u01/app/oracle/product/10.2.0/db_1" ORACLE_HOME_NAME="home1"

It asked to run root.sh as root.

Thursday, September 2, 2010

Open database with inconsistent data

The backup script was accidentally deleted by SA. We don't have the backup and the customer require a recovery to a certain date. The only backup we have is an operating system backup and all archive logs. I started with a backup and applied archivelogs for about 2 days. But no matter how many archive logs applied, I still got the following message:

Specify log: {=suggested filename AUTO CANCEL}
cancel
Media recovery cancelled.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'E:\ORACLE\ORADATA\ASMP\SYSTEM01.DBF'

Solution:

at init file set _ALLOW_RESETLOGS_CORRUPTION=TRUE and then the database is opened.
Do a full export of the database, create a new one and do an import.

SQL> startup mount;ORACLE instance started.
Total System Global Area 453498908 bytesFixed Size 75804 bytesVariable Size 295604224 bytesDatabase Buffers 157286400 bytesRedo Buffers 532480 bytesDatabase mounted.SQL> recover database using backup controlfile until cancel;ORA-00279: change 754753809 generated at 08/24/2010 12:15:12 needed for thread1ORA-00289: suggestion : E:\ORACLE\ORADATA\ASMP\ARCHIVE\ARC82677.001ORA-00280: change 754753809 for thread 1 is in sequence #382677
Specify log: {=suggested filename AUTO CANCEL}
ORA-00279: change 754755854 generated at 08/24/2010 12:15:53 needed for thread1ORA-00289: suggestion : E:\ORACLE\ORADATA\ASMP\ARCHIVE\ARC82678.001ORA-00280: change 754755854 for thread 1 is in sequence #382678ORA-00278: log file 'E:\ORACLE\ORADATA\ASMP\ARCHIVE\ARC82677.001' no longerneeded for this recovery
Specify log: {=suggested filename AUTO CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistentORA-01110: data file 1: 'E:\ORACLE\ORADATA\ASMP\SYSTEM01.DBF'
ORA-01112: media recovery not started
SQL> alter database open resetlogs;
Database altered.
SQL>

Tuesday, August 31, 2010

Instructions to download Cobol 5 for PeopleSoft

* Go to E-Delivery, http://edelivery.oracle.com, and login by providing the requested information.

* On the search page, select the following from the drop down menus:
o Select a Product Pack = PeopleSoft Enterprise
o Platform = Linux x86

* On the search results page, scroll down and click on the Third Party – Micro Focus 5.0 for PeopleSoft Enterprise Media Pack link.

* On the download components page, download the following files associated with Server Express for Linux x86.
o V14660-01, PeopleSoft Enterprise COBOL Quick Reference Guide - Version 9
o B45429-01, Micro Focus Server Express 5.0 Wrap Pack 4 Extras Install Documentation for PeopleSoft and Runtime Licenses (Read "Readme" file located on this disk for instructions) (CD)
o B45428-01, Micro Focus Server Express 5.0 Wrap Pack 4 for PeopleSoft Documentation CD
o B45424-01, Micro Focus Server Express 5.0 Wrap Pack 4 for PeopleSoft for Red Hat Linux and SuSE Linux (CD) (installation files)

* Please note that the installation CD (B45424-01) contains .tar files which are the correct file extension for Linux install files.

Sunday, August 29, 2010

ORA-16014: log 4 sequence# 20 not archived, no available destinations

RAC environment with ASM.

While try to open the database, got the following error:

SQL> alter database open;alter database open*ERROR at line 1:ORA-16014: log 4 sequence# 20 not archived, no available destinationsORA-00312: online log 4 thread 2:'+PASS_DATA/upstrprd_prime/onlinelog/group_4.266.720285169'ORA-00312: online log 4 thread 2:'+PASS_FLASH/upstrprd_prime/onlinelog/group_4.260.720285169'

Looks like some online log files are corrupted.

The solution:

SQL> alter database clear unarchived logfile group 4;
Database altered.
SQL> alter database open;alter database open*ERROR at line 1:ORA-16014: log 3 sequence# 21 not archived, no available destinationsORA-00312: online log 3 thread 2:'+PASS_DATA/upstrprd_prime/onlinelog/group_3.265.720285169'ORA-00312: online log 3 thread 2:'+PASS_FLASH/upstrprd_prime/onlinelog/group_3.259.720285169'
SQL> alter database clear unarchived logfile group 3;
Database altered.
SQL> alter database open;
Database altered.
SQL>

Perform a backup after open the database.

Wednesday, August 25, 2010

PeopleTools 8.49 installation on ZLinux

JAVA 6 is installed.

Got the following errors when installing:

Verification failed for /opt/ibm/java-s390x-60 using the JVM file /tmp/istemp19471236113223/sunjdk15.jvm
Verifying... /opt/ibm/java-s390x-60/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.vendor java.version
Verification failed for /opt/ibm/java-s390x-60 using the JVM file /tmp/istemp19471236113223/sunjre15.jvm
JVM specified with -is:javahome cannot be verified with the JVM files specified with the launcher. Environment Variables will be checked next...
Checking the environment variables specifed in the JVM files to find the JVM...
Verifying... /usr/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.version
$JDK_HOME is not defined in the shell environment specifed in the JVM FILE sunjdk15.jvm
$JAVAHOME is not defined in the shell environment specifed in the JVM FILE sunjdk15.jvm
A jvm found at /opt/ibm/java-s390x-60/bin/java
Verifying... /opt/ibm/java-s390x-60/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.version
Verification failed for /opt/ibm/java-s390x-60/bin/java using the JVM file /tmp/istemp19471236113223/sunjdk15.jvm.
Verifying... /usr/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.vendor java.version
$JDK_HOME is not defined in the shell environment specifed in the JVM FILE sunjre15.jvm
$JAVAHOME is not defined in the shell environment specifed in the JVM FILE sunjre15.jvm
A jvm found at /opt/ibm/java-s390x-60/bin/java
Verifying... /opt/ibm/java-s390x-60/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.vendor java.version
Verification failed for /opt/ibm/java-s390x-60/bin/java using the JVM file /tmp/istemp19471236113223/sunjre15.jvm.
No JVM can be found using the shell environment variable. Searching JVM will continue with Path Hints specified in the JVM Files...
--------------------------------------------------------------------------------------------
Searching a JVM using /tmp/istemp19471236113223/sunjdk15.jvm
Verifying... /opt/IBMJava2-s390-142/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.version
Verification failed for /opt/IBMJava2-s390-142/bin/java using the JVM file /tmp/istemp19471236113223/sunjdk15.jvm.
Verifying... /opt/IBMJava2-s390x-142/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.version
Verification failed for /opt/IBMJava2-s390x-142/bin/java using the JVM file /tmp/istemp19471236113223/sunjdk15.jvm.
Verifying... /opt/IBMJava2-s390-142/jre/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.version
Verification failed for /opt/IBMJava2-s390-142/jre/bin/java using the JVM file /tmp/istemp19471236113223/sunjdk15.jvm.
Verifying... /opt/IBMJava2-s390x-142/jre/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.version
Verification failed for /opt/IBMJava2-s390x-142/jre/bin/java using the JVM file /tmp/istemp19471236113223/sunjdk15.jvm.
Verifying... /opt/IBMJava2-s390-142/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.version
Verification failed for /opt/IBMJava2-s390-142/bin/java using the JVM file /tmp/istemp19471236113223/sunjdk15.jvm.
Verifying... /opt/IBMJava2-s390x-142/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.version
Verification failed for /opt/IBMJava2-s390x-142/bin/java using the JVM file /tmp/istemp19471236113223/sunjdk15.jvm.
--------------------------------------------------------------------------------------------
Searching a JVM using /tmp/istemp19471236113223/sunjre15.jvm
Verifying... /opt/IBMJava2-s390-142/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.vendor java.version
Verification failed for /opt/IBMJava2-s390-142/bin/java using the JVM file /tmp/istemp19471236113223/sunjre15.jvm.
Verifying... /opt/IBMJava2-s390x-142/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.vendor java.version
Verification failed for /opt/IBMJava2-s390x-142/bin/java using the JVM file /tmp/istemp19471236113223/sunjre15.jvm.
Verifying... /opt/IBMJava2-s390-142/jre/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.vendor java.version
Verification failed for /opt/IBMJava2-s390-142/jre/bin/java using the JVM file /tmp/istemp19471236113223/sunjre15.jvm.
Verifying... /opt/IBMJava2-s390x-142/jre/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.vendor java.version
Verification failed for /opt/IBMJava2-s390x-142/jre/bin/java using the JVM file /tmp/istemp19471236113223/sunjre15.jvm.
Verifying... /opt/IBMJava2-s390-142/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.vendor java.version
Verification failed for /opt/IBMJava2-s390-142/bin/java using the JVM file /tmp/istemp19471236113223/sunjre15.jvm.
Verifying... /opt/IBMJava2-s390x-142/bin/java -cp /tmp/istemp19471236113223/Verify.jar Verify java.vendor java.version
Verification failed for /opt/IBMJava2-s390x-142/bin/java using the JVM file /tmp/istemp19471236113223/sunjre15.jvm.
Extracting bundled JRE...
checking disk space on the parition /tmp/istemp19471236113223 for /tmp/istemp19471236113223/linux-sun-1.5.0.bin
60962 512 bytes disk blocks required.
Cheking for disk quotas...
USER=mukeshb
LINE=
WORDS=0
Disk quota command not enabled.
44082520 512 bytes disk blocks available on the partition /tmp/istemp19471236113223
Disk space check on the parition /tmp/istemp19471236113223 succeeded.
checking disk space on the parition /tmp/istemp19471236113223 for /tmp/istemp19471236113223/linux-sun-1.5.0.bin
Bundled JRE is not binary compatible with host OS/Arch or it is corrupt. Testing bundled JRE failed.

Solution: install JAVA 5.

Data pump problem ORA-39071: Value for EXCLUDE is badly formed.

I tried to use datapump to import a whole database but I want to include sys and system. I specified exclude=schema:sys,system and got the following errors:

Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing options

ORA-39001: invalid argument value
ORA-39071: Value for EXCLUDE is badly formed.
ORA-00920: invalid relational operator
ORA-06512: at "SYS.KUPM$MCP", line 2687


Solution: "exclude=schema:\"='SYS'\"" "exclude=schema:\"='SYSTEM'\""

Reference Oracle note 734324.1 .

Friday, July 30, 2010

RMAN-06136: ORACLE error from auxiliary database: ORA-19563: header validation failed for file

I have an script to refresh databases every night. This morning, it failed with error


RMAN-06136: ORACLE error from auxiliary database: ORA-19563: header validation failed for file .

No scripts modification recently. By a little research, it's been found out that another DBA added a data file with the same name on the production database. The file been added didn't cause any problem in the production database since it's added to a different mounting point. But the when refreshing, it maps to the same mounting point. As a result, the file got overwritten.

The solution is obviously to change the name at the source database. But since it's production, we can not do it without asking down time. So the work around is to add the following line to the init file of the database to be refreshed.

'/u09/oradata/HCMPROD/hrapp2.dbf','/u15/oradata/hcmstg/hrapp100.dbf',

Grid Control issue to configure a database in a different domain

Grid Control version: 10.2.0.5.

The repository domain is dc.gov but the database in question is located at in.dc.gov. When configuring the database, got the following error:

oracle.sysman.emSDK.emd.comm.CommException: java.net.SocketException: Connection timed out:could be due to invalid address

It's resolved after SA did something for the network. Here's what's been done.


search in.dc.gov dc.gov
nameserver 10.201.1.241
nameserver 10.201.1.242

Looks like the existing nameservers are not good enough.

Wednesday, July 28, 2010

Z/Linux Oracle 10gR2 installation reference

There's a document about recommended articles on database installation on Z/Linux.
Recommended metalink articles for Oracle Database 10gR2 installation on zLinux (s390x) [ID 761261.1]

Here are some of the recommendations:

General: Note 415182.1 - DB Install Requirements Quick Reference - zSeries based Linux
Note 851598.1 - Linux x86, x86-64, and s390x Requirements Reference List
Red Hat Enterprise Linux (RHEL):
Note 376183.1 - Defining a "default RPMs" installation of the RHEL OS
Note 420382.1 - Requirements for Installing Oracle 10gR2 RDBMS on RHEL 4 on zLinux (s390x)
Suse Linux Enterprise Server (SLES):
Note 386391.1 - Defining a "default RPMs" installation of the SLES OS
Note 431443.1 - Requirements for Installing Oracle 10gR2 RDBMS on SLES 9 zLinux (s390x)Note 471165.1 - Additional steps to install 10gR2 RAC on IBM zSeries Based Linux (SLES10)Note 556906.1 - Requirements for Installing Oracle 10gR2 RDBMS on SLES 10 zLinux (s390x)

References

NOTE:851598.1 - Linux OS Requirements Reference List for Database Server
NOTE:415182.1 - DB Install Requirements Quick Reference - zSeries based Linux
NOTE:420382.1 - Requirements for Installing Oracle 10gR2 RDBMS on RHEL 4 on zLinux (s390x)
NOTE:431443.1 - Requirements for Installing Oracle 10gR2 RDBMS on SLES 9 zLinux (s390x)NOTE:471165.1 - Additional steps to install 10gR2 RAC on IBM zSeries Based Linux (SLES10)NOTE:556906.1 - Requirements for Installing Oracle 10gR2 RDBMS on SLES 10 zLinux (s390x)
NOTE:741646.1 - Requirements for Installing Oracle 10gR2 RDBMS on RHEL 5 on zLinux (s390x)

Ensure you have prerequisite rpms to install Oracle Server 10gR2 on Linux on IBM zSeries [ID 1086769.1]

Requirements for Installing Oracle 10gR2 RDBMS on RHEL 5 on zLinux (s390x) [ID 741646.1]

Can not start dbca and sqlplus after 10gR2 installation on Z/Linux

After installed Oracle 10g and patched it to 10.2.0.4. I'm trying to create a database and getting the following errors:

[oracle@zLinux70 ~]$ dbcaWarning: JIT compiler "jitc" not found. Will use interpreter.UnsatisfiedLinkError exception loading native library: njni10Exception in thread "main" java.lang.UnsatisfiedLinkError: get at oracle.net.config.Config.getNetDir(Unknown Source) at oracle.net.config.Config.initConfig(Unknown Source) at oracle.net.config.Config.(Unknown Source) at oracle.sysman.assistants.util.NetworkUtils.(NetworkUtils.java:225) at oracle.sysman.assistants.util.step.StepContext.(StepContext.java:264) at oracle.sysman.assistants.dbca.backend.Host.(Host.java:686) at oracle.sysman.assistants.dbca.ui.UIHost.(UIHost.java:205) at oracle.sysman.assistants.dbca.ui.InteractiveHost.(InteractiveHost.java:54) at oracle.sysman.assistants.dbca.Dbca.getHost(Dbca.java:160) at oracle.sysman.assistants.dbca.Dbca.execute(Dbca.java:94) at oracle.sysman.assistants.dbca.Dbca.main(Dbca.java:180)

Then I tried sqlplus and also it's not working.
[oracle@zLinux70 ~]$ sqlplus /nologsqlplus: error while loading shared libraries: /u01/app/oracle/oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1: cannot restore segment prot after reloc: Permission denied

By searching on Oracle support, I found the document
DBCA fails: UnsatisfiedLinkError exception loading native library: njni10 in z/Linux [ID 875013.1]. It says to disable SELinux.

Solution
Modify /etc/selinux/config file to set SELINUX=disabled as follows:

SELINUX=disabled.
SELINUXTYPE=targeted

The Linux admin had to bounce the server after the change.

[oracle@zLinux70 ~]$ /usr/sbin/sestatusSELinux status: disabled
[oracle@zLinux70 ~]$ /usr/sbin/getenforceDisabled

The issue is resolved after the bounce.

Tuesday, July 27, 2010

Oracle 10gR2 installation on Z/Linux 5

I'm installing Oracle 10gR2 on Z/Linux 5.

While starting the installer, got the following errors:
JVMDG080: Cannot find class com/ibm/jvm/Trace
JVMXM012: Error occurred in final diagnostics initialization
Could not create the Java virtual machine.

Did export JAVA_COMPILER=NONE according to document 400785.1 then started it again. Now got a different error:

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-07-26_04-13-20PM. Please wait ...[oracle@zLinux70 database]$ The java class could not be loaded. java.lang.ExceptionInInitializerError

Then I tried the flag -ignoreSysPrereqs, and got the following errors:
Exception java.lang.ExceptionInInitializerError occurred..java.lang.ExceptionInInitializerError at java.awt.Component.(Component.java:548) at oracle.sysman.oii.oiif.oiifm.OiifmGraphicInterfaceManager.(OiifmGraphicInterfaceManager.java:222) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.createInterfaceManager(OiicSessionInterfaceManager.java:194) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.getInterfaceManager(OiicSessionInterfaceManager.java:203) at oracle.sysman.oii.oiic.OiicInstaller.getInterfaceManager(OiicInstaller.java:437) at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:932) at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:872)Caused by: java.lang.NullPointerException at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2171) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2006) at java.lang.Runtime.loadLibrary0(Runtime.java:824) at java.lang.System.loadLibrary(System.java:910) at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:76) at java.security.AccessController.doPrivileged1(Native Method) at java.security.AccessController.doPrivileged(AccessController.java:287) at sun.awt.NativeLibLoader.loadLibraries(NativeLibLoader.java:81) at sun.awt.DebugHelper.(DebugHelper.java:56) ... 7 moreException in thread "main" java.lang.NoClassDefFoundError: oracle/sysman/oii/oiif/oiifm/OiifmMainFrame at oracle.sysman.oii.oiif.oiifm.OiifmGraphicInterfaceManager.(OiifmGraphicInterfaceManager.java:222) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.createInterfaceManager(OiicSessionInterfaceManager.java:194) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.getInterfaceManager(OiicSessionInterfaceManager.java:203) at oracle.sysman.oii.oiif.oiifm.OiifmAlert.(OiifmAlert.java:151) at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:990) at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:872)

It's working after run the verification utility supplied by Oracle. It's documented as
Ensure you have prerequisite rpms to install Oracle Server 10gR2 on Linux on IBM zSeries [ID 1086769.1]

[ServerOps1@zLinux70 software]$ sudo rpm -Uvh ora-val-rpm-EL5-DB-10.2.0.4-1.s390x.rpm
Preparing... ########################################### [100%]
1:ora-val-rpm-EL5-DB ########################################### [100%]
*************************************************************************************
* Validation complete - please install any missing packages *
* Note that these packages cannot be fully validated automatically: *
* glibc-dev *
* libaio *
* Please check the output of the following command *
* rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep glibc-dev *
* rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep libaio *
* which should return output similar to the following: - *
* *
* glibc-devel-2.5-24 (s390x) *
* glibc-devel-2.5-24 (s390) *
* *
* libaio-0.3.106-3.2 (s390x) *
* libaio-0.3.106-3.2 (s390) *
*************************************************************************************
glibc-devel-2.5-49.el5_5.4 (s390x)
glibc-devel-2.5-49.el5_5.4 (s390)
libaio-devel-0.3.106-3.2 (s390)
libaio-0.3.106-3.2 (s390x)
libaio-0.3.106-3.2 (s390)
libaio-devel-0.3.106-3.2 (s390x)
[ServerOps1@zLinux70 software]$ rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep glibc-dev
glibc-devel-2.5-49.el5_5.4 (s390x)
glibc-devel-2.5-49.el5_5.4 (s390)
[ServerOps1@zLinux70 software]$ rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep libaio
libaio-devel-0.3.106-3.2 (s390)
libaio-0.3.106-3.2 (s390x)
libaio-0.3.106-3.2 (s390)
libaio-devel-0.3.106-3.2 (s390x)
[ServerOps1@zLinux70 software]$

Monday, July 26, 2010

Bug : ORA-19921: MAXIMUM NUMBER OF 128 ROWS EXCEEDED CONNECTING TO TARGET

While connecting to target database via RMAN, got the following errors:

RMAN-6900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT rowRMAN-6901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rowsORACLE error from target database:ORA-19921: maximum number of 128 rows exceeded.

By searching, there's an Oracle bug filed. Bug number is Bug 8461724.

But the official Oracle document of the bug says:

Workaround: no.
It's fixed in 11g release 2.

Upgrading a database from 10g to 11g is a project takes months because testing has to be performed before production upgrade. Here is how it's resolved. It might not apply to your case.

SQL> select sid, serial#, program from v$session where program like '%rman%';
SID SERIAL# PROGRAM---------- ---------- ------------------------------------------------ 272 10130 rman@asmpu33 (TNS V1-V3) 353 20880 rman@asmpu33 (TNS V1-V3)
SQL> alter system kill session '272,10130';
System altered.
SQL> alter system kill session '353,20880';
System altered.


By killing the two rman sessions, the issue is resolved.

Thursday, February 11, 2010

One example of internal error

One of my collegue called me for the following internal error on Oracle 10g R2:

5202,5270d5201
< 382952:ORA-27300: OS system dependent operation:fork failed with status: 17
< 382953:ORA-27301: OS failure message: File exists
< 382954:ORA-27302: failure occurred at: skgpspawn5
< 382955:ORA-27303: additional information: skgpspawn5
< 382987:ORA-27300: OS system dependent operation:fork failed with status: 2
< 382988:ORA-27301: OS failure message: No such file or directory
< 382989:ORA-27302: failure occurred at: skgpspawn5
< 382990:ORA-27303: additional information: skgpspawn5
< 382995:ORA-00443: background process "ARC0" did not start
< 383000:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383001:ORA-27301: OS failure message: No such file or directory
< 383002:ORA-27302: failure occurred at: skgpspawn5
< 383003:ORA-27303: additional information: skgpspawn5
< 383008:ORA-00443: background process "ARC1" did not start
< 383013:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383014:ORA-27301: OS failure message: No such file or directory
< 383015:ORA-27302: failure occurred at: skgpspawn5
< 383016:ORA-27303: additional information: skgpspawn5
< 383021:ORA-00443: background process "ARC2" did not start
< 383026:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383027:ORA-27301: OS failure message: No such file or directory
< 383028:ORA-27302: failure occurred at: skgpspawn5
< 383029:ORA-27303: additional information: skgpspawn5
< 383034:ORA-00443: background process "ARC3" did not start
< 383039:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383040:ORA-27301: OS failure message: No such file or directory
< 383041:ORA-27302: failure occurred at: skgpspawn5
< 383042:ORA-27303: additional information: skgpspawn5
< 383047:ORA-00443: background process "ARC4" did not start
< 383052:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383053:ORA-27301: OS failure message: No such file or directory
< 383054:ORA-27302: failure occurred at: skgpspawn5
< 383055:ORA-27303: additional information: skgpspawn5
< 383060:ORA-00443: background process "ARC5" did not start
< 383065:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383066:ORA-27301: OS failure message: No such file or directory
< 383067:ORA-27302: failure occurred at: skgpspawn5
< 383068:ORA-27303: additional information: skgpspawn5
< 383073:ORA-00443: background process "ARC6" did not start
< 383078:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383079:ORA-27301: OS failure message: No such file or directory
< 383080:ORA-27302: failure occurred at: skgpspawn5
< 383081:ORA-27303: additional information: skgpspawn5
< 383086:ORA-00443: background process "ARC7" did not start
< 383091:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383092:ORA-27301: OS failure message: No such file or directory
< 383093:ORA-27302: failure occurred at: skgpspawn5
< 383094:ORA-27303: additional information: skgpspawn5
< 383099:ORA-00443: background process "ARC8" did not start
< 383104:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383105:ORA-27301: OS failure message: No such file or directory
< 383106:ORA-27302: failure occurred at: skgpspawn5
< 383107:ORA-27303: additional information: skgpspawn5
< 383112:ORA-00443: background process "ARC9" did not start
< 383117:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383118:ORA-27301: OS failure message: No such file or directory
< 383119:ORA-27302: failure occurred at: skgpspawn5
< 383120:ORA-27303: additional information: skgpspawn5
< 383125:ORA-00443: background process "ARCa" did not start
< 383130:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383131:ORA-27301: OS failure message: No such file or directory
< 383132:ORA-27302: failure occurred at: skgpspawn5
< 383133:ORA-27303: additional information: skgpspawn5
< 383138:ORA-00443: background process "ARCb" did not start
< 383148:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383149:ORA-27301: OS failure message: No such file or directory
< 383150:ORA-27302: failure occurred at: skgpspawn5
< 383151:ORA-27303: additional information: skgpspawn5
< 383156:ORA-00443: background process "" did not start

It our production database and it can not be started. Finally I figured out the cause is that the database was not shutdown properly. There're some background processes still running when he tried to start the database. After killing the running processes, the database were able to start.