Can not start dbca and sqlplus after 10gR2 installation on Z/Linux

After installed Oracle 10g and patched it to 10.2.0.4. I'm trying to create a database and getting the following errors:

[oracle@zLinux70 ~]$ dbcaWarning: JIT compiler "jitc" not found. Will use interpreter.UnsatisfiedLinkError exception loading native library: njni10Exception in thread "main" java.lang.UnsatisfiedLinkError: get at oracle.net.config.Config.getNetDir(Unknown Source) at oracle.net.config.Config.initConfig(Unknown Source) at oracle.net.config.Config.(Unknown Source) at oracle.sysman.assistants.util.NetworkUtils.(NetworkUtils.java:225) at oracle.sysman.assistants.util.step.StepContext.(StepContext.java:264) at oracle.sysman.assistants.dbca.backend.Host.(Host.java:686) at oracle.sysman.assistants.dbca.ui.UIHost.(UIHost.java:205) at oracle.sysman.assistants.dbca.ui.InteractiveHost.(InteractiveHost.java:54) at oracle.sysman.assistants.dbca.Dbca.getHost(Dbca.java:160) at oracle.sysman.assistants.dbca.Dbca.execute(Dbca.java:94) at oracle.sysman.assistants.dbca.Dbca.main(Dbca.java:180)

Then I tried sqlplus and also it's not working.
[oracle@zLinux70 ~]$ sqlplus /nologsqlplus: error while loading shared libraries: /u01/app/oracle/oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1: cannot restore segment prot after reloc: Permission denied

By searching on Oracle support, I found the document
DBCA fails: UnsatisfiedLinkError exception loading native library: njni10 in z/Linux [ID 875013.1]. It says to disable SELinux.

Solution
Modify /etc/selinux/config file to set SELINUX=disabled as follows:

SELINUX=disabled.
SELINUXTYPE=targeted

The Linux admin had to bounce the server after the change.

[oracle@zLinux70 ~]$ /usr/sbin/sestatusSELinux status: disabled
[oracle@zLinux70 ~]$ /usr/sbin/getenforceDisabled

The issue is resolved after the bounce.

Oracle 10gR2 installation on Z/Linux 5

I'm installing Oracle 10gR2 on Z/Linux 5.

While starting the installer, got the following errors:
JVMDG080: Cannot find class com/ibm/jvm/Trace
JVMXM012: Error occurred in final diagnostics initialization
Could not create the Java virtual machine.

Did export JAVA_COMPILER=NONE according to document 400785.1 then started it again. Now got a different error:

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-07-26_04-13-20PM. Please wait ...[oracle@zLinux70 database]$ The java class could not be loaded. java.lang.ExceptionInInitializerError

Then I tried the flag -ignoreSysPrereqs, and got the following errors:
Exception java.lang.ExceptionInInitializerError occurred..java.lang.ExceptionInInitializerError at java.awt.Component.(Component.java:548) at oracle.sysman.oii.oiif.oiifm.OiifmGraphicInterfaceManager.(OiifmGraphicInterfaceManager.java:222) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.createInterfaceManager(OiicSessionInterfaceManager.java:194) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.getInterfaceManager(OiicSessionInterfaceManager.java:203) at oracle.sysman.oii.oiic.OiicInstaller.getInterfaceManager(OiicInstaller.java:437) at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:932) at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:872)Caused by: java.lang.NullPointerException at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2171) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2006) at java.lang.Runtime.loadLibrary0(Runtime.java:824) at java.lang.System.loadLibrary(System.java:910) at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:76) at java.security.AccessController.doPrivileged1(Native Method) at java.security.AccessController.doPrivileged(AccessController.java:287) at sun.awt.NativeLibLoader.loadLibraries(NativeLibLoader.java:81) at sun.awt.DebugHelper.(DebugHelper.java:56) ... 7 moreException in thread "main" java.lang.NoClassDefFoundError: oracle/sysman/oii/oiif/oiifm/OiifmMainFrame at oracle.sysman.oii.oiif.oiifm.OiifmGraphicInterfaceManager.(OiifmGraphicInterfaceManager.java:222) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.createInterfaceManager(OiicSessionInterfaceManager.java:194) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.getInterfaceManager(OiicSessionInterfaceManager.java:203) at oracle.sysman.oii.oiif.oiifm.OiifmAlert.(OiifmAlert.java:151) at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:990) at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:872)

It's working after run the verification utility supplied by Oracle. It's documented as
Ensure you have prerequisite rpms to install Oracle Server 10gR2 on Linux on IBM zSeries [ID 1086769.1]

[ServerOps1@zLinux70 software]$ sudo rpm -Uvh ora-val-rpm-EL5-DB-10.2.0.4-1.s390x.rpm
Preparing... ########################################### [100%]
1:ora-val-rpm-EL5-DB ########################################### [100%]
*************************************************************************************
* Validation complete - please install any missing packages *
* Note that these packages cannot be fully validated automatically: *
* glibc-dev *
* libaio *
* Please check the output of the following command *
* rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep glibc-dev *
* rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep libaio *
* which should return output similar to the following: - *
* *
* glibc-devel-2.5-24 (s390x) *
* glibc-devel-2.5-24 (s390) *
* *
* libaio-0.3.106-3.2 (s390x) *
* libaio-0.3.106-3.2 (s390) *
*************************************************************************************
glibc-devel-2.5-49.el5_5.4 (s390x)
glibc-devel-2.5-49.el5_5.4 (s390)
libaio-devel-0.3.106-3.2 (s390)
libaio-0.3.106-3.2 (s390x)
libaio-0.3.106-3.2 (s390)
libaio-devel-0.3.106-3.2 (s390x)
[ServerOps1@zLinux70 software]$ rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep glibc-dev
glibc-devel-2.5-49.el5_5.4 (s390x)
glibc-devel-2.5-49.el5_5.4 (s390)
[ServerOps1@zLinux70 software]$ rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep libaio
libaio-devel-0.3.106-3.2 (s390)
libaio-0.3.106-3.2 (s390x)
libaio-0.3.106-3.2 (s390)
libaio-devel-0.3.106-3.2 (s390x)
[ServerOps1@zLinux70 software]$

Bug : ORA-19921: MAXIMUM NUMBER OF 128 ROWS EXCEEDED CONNECTING TO TARGET

While connecting to target database via RMAN, got the following errors:

RMAN-6900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT rowRMAN-6901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rowsORACLE error from target database:ORA-19921: maximum number of 128 rows exceeded.

By searching, there's an Oracle bug filed. Bug number is Bug 8461724.

But the official Oracle document of the bug says:

Workaround: no.
It's fixed in 11g release 2.

Upgrading a database from 10g to 11g is a project takes months because testing has to be performed before production upgrade. Here is how it's resolved. It might not apply to your case.

SQL> select sid, serial#, program from v$session where program like '%rman%';
SID SERIAL# PROGRAM---------- ---------- ------------------------------------------------ 272 10130 rman@asmpu33 (TNS V1-V3) 353 20880 rman@asmpu33 (TNS V1-V3)
SQL> alter system kill session '272,10130';
System altered.
SQL> alter system kill session '353,20880';
System altered.


By killing the two rman sessions, the issue is resolved.

One example of internal error

One of my collegue called me for the following internal error on Oracle 10g R2:

5202,5270d5201
< 382952:ORA-27300: OS system dependent operation:fork failed with status: 17
< 382953:ORA-27301: OS failure message: File exists
< 382954:ORA-27302: failure occurred at: skgpspawn5
< 382955:ORA-27303: additional information: skgpspawn5
< 382987:ORA-27300: OS system dependent operation:fork failed with status: 2
< 382988:ORA-27301: OS failure message: No such file or directory
< 382989:ORA-27302: failure occurred at: skgpspawn5
< 382990:ORA-27303: additional information: skgpspawn5
< 382995:ORA-00443: background process "ARC0" did not start
< 383000:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383001:ORA-27301: OS failure message: No such file or directory
< 383002:ORA-27302: failure occurred at: skgpspawn5
< 383003:ORA-27303: additional information: skgpspawn5
< 383008:ORA-00443: background process "ARC1" did not start
< 383013:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383014:ORA-27301: OS failure message: No such file or directory
< 383015:ORA-27302: failure occurred at: skgpspawn5
< 383016:ORA-27303: additional information: skgpspawn5
< 383021:ORA-00443: background process "ARC2" did not start
< 383026:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383027:ORA-27301: OS failure message: No such file or directory
< 383028:ORA-27302: failure occurred at: skgpspawn5
< 383029:ORA-27303: additional information: skgpspawn5
< 383034:ORA-00443: background process "ARC3" did not start
< 383039:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383040:ORA-27301: OS failure message: No such file or directory
< 383041:ORA-27302: failure occurred at: skgpspawn5
< 383042:ORA-27303: additional information: skgpspawn5
< 383047:ORA-00443: background process "ARC4" did not start
< 383052:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383053:ORA-27301: OS failure message: No such file or directory
< 383054:ORA-27302: failure occurred at: skgpspawn5
< 383055:ORA-27303: additional information: skgpspawn5
< 383060:ORA-00443: background process "ARC5" did not start
< 383065:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383066:ORA-27301: OS failure message: No such file or directory
< 383067:ORA-27302: failure occurred at: skgpspawn5
< 383068:ORA-27303: additional information: skgpspawn5
< 383073:ORA-00443: background process "ARC6" did not start
< 383078:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383079:ORA-27301: OS failure message: No such file or directory
< 383080:ORA-27302: failure occurred at: skgpspawn5
< 383081:ORA-27303: additional information: skgpspawn5
< 383086:ORA-00443: background process "ARC7" did not start
< 383091:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383092:ORA-27301: OS failure message: No such file or directory
< 383093:ORA-27302: failure occurred at: skgpspawn5
< 383094:ORA-27303: additional information: skgpspawn5
< 383099:ORA-00443: background process "ARC8" did not start
< 383104:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383105:ORA-27301: OS failure message: No such file or directory
< 383106:ORA-27302: failure occurred at: skgpspawn5
< 383107:ORA-27303: additional information: skgpspawn5
< 383112:ORA-00443: background process "ARC9" did not start
< 383117:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383118:ORA-27301: OS failure message: No such file or directory
< 383119:ORA-27302: failure occurred at: skgpspawn5
< 383120:ORA-27303: additional information: skgpspawn5
< 383125:ORA-00443: background process "ARCa" did not start
< 383130:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383131:ORA-27301: OS failure message: No such file or directory
< 383132:ORA-27302: failure occurred at: skgpspawn5
< 383133:ORA-27303: additional information: skgpspawn5
< 383138:ORA-00443: background process "ARCb" did not start
< 383148:ORA-27300: OS system dependent operation:fork failed with status: 2
< 383149:ORA-27301: OS failure message: No such file or directory
< 383150:ORA-27302: failure occurred at: skgpspawn5
< 383151:ORA-27303: additional information: skgpspawn5
< 383156:ORA-00443: background process "" did not start

It our production database and it can not be started. Finally I figured out the cause is that the database was not shutdown properly. There're some background processes still running when he tried to start the database. After killing the running processes, the database were able to start.

Relocate datafiles for physical dataguard (Oracle 10.2)

Relocate datafiles for physical dataguard (Oracle 10.2)

We will relocate the following files at a physical data guard situation.

File Move file to
/u02/oradata/HRPRD/hrapp.dbf /u07
/u03/oradata/HRPRD/psundo03.dbf /u06
/u05/oradata/HRPRD/psindex04.dbf /u04

hrapp.dbf and psindex04.dbf are regular data files. Since psundo03.dbf is a data file in undo table space, it takes more steps.

1. Rename files at primary

SQL> col file_name format a40
SQL> select file_name, tablespace_name from dba_data_files where file_name like '%hrapp.dbf';

FILE_NAME TABLESPACE_NAME
---------------------------------------- ------------------------------
/u02/oradata/HRPRD/hrapp.dbf HRAPP

SQL> select file_name, tablespace_name from dba_data_files where file_name like '%psundo03.dbf';

FILE_NAME TABLESPACE_NAME
---------------------------------------- ------------------------------
/u03/oradata/HRPRD/psundo03.dbf PSUNDO

SQL> select file_name, tablespace_name from dba_data_files where file_name like '%psindex04.dbf';

FILE_NAME TABLESPACE_NAME
---------------------------------------- ------------------------------
/u05/oradata/HRPRD/psindex04.dbf PSINDEX

SQL> alter tablespace HRAPP offline;

Tablespace altered.

SQL> !mv /u02/oradata/HRPRD/hrapp.dbf /u07/oradata/HRPRD/hrapp.dbf

SQL> alter tablespace HRAPP rename datafile '/u02/oradata/HRPRD/hrapp.dbf' to '/u07/oradata/HRPRD/hrapp.dbf';

Tablespace altered.

SQL> alter tablespace HRAPP online;

Tablespace altered.

2. Connect to standby database

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> shutdown;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> ho
$ mv /u02/oradata/HRPRD/hrapp.dbf /u07/oradata/HRPRD/hrapp.dbf
$ exit

SQL> startup mount;
ORACLE instance started.

Total System Global Area 570425344 bytes
Fixed Size 2072872 bytes
Variable Size 293605080 bytes
Database Buffers 268435456 bytes
Redo Buffers 6311936 bytes
Database mounted.

SQL> alter system set standby_file_management=MANUAL;

System altered.

SQL> alter database rename file '/u02/oradata/HRPRD/hrapp.dbf' to '/u07/oradata/HRPRD/hrapp.dbf';

Database altered.

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL>

The relocation of /u05/oradata/HRPRD/psindex04.dbf is the same as above.

3. Working on undo table space

Beside the steps for regular table space, we have to create a new undo table space undo02, then switch the current undo table space to undo02. After we finished the relocation, we drop undo02.
Primary side:

SQL> CREATE UNDO TABLESPACE undo02 DATAFILE '/u06/oradata/HRPRD/undo0201.dbf' SIZE 2048M AUTOEXTEND ON;


Tablespace created.

SQL> show parameter undo

NAME TYPE VALUE
------------------------------------ -------------------------------- ------------------------------
undo_management string AUTO
undo_retention integer 1800
undo_tablespace string psundo
SQL> alter system set undo_tablespace=undo02;

System altered.

SQL> alter tablespace psundo offline;

Tablespace altered.

SQL> !mv /u03/oradata/HRPRD/psundo03.dbf /u06/oradata/HRPRD/psundo03.dbf

SQL> alter tablespace psundo rename datafile '/u03/oradata/HRPRD/psundo03.dbf' to '/u06/oradata/HRPRD/psundo03.dbf';

Tablespace altered.

SQL> alter tablespace psundo online;

Tablespace altered.

SQL> alter system set undo_tablespace=psundo;

System altered.

SQL> drop tablespace undo02;

Tablespace dropped.

SQL>

Standby side:


SQL> alter database recover managed standby database cancel;

Database altered.

SQL> shutdown;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> !mv /u03/oradata/HRPRD/psundo03.dbf /u06/oradata/HRPRD/psundo03.dbf

SQL> startup mount;
ORACLE instance started.

Total System Global Area 570425344 bytes
Fixed Size 2072872 bytes
Variable Size 293605080 bytes
Database Buffers 268435456 bytes
Redo Buffers 6311936 bytes
Database mounted.
SQL> show parameter standby

NAME TYPE
------------------------------------ --------------------------------
VALUE
------------------------------
standby_archive_dest string
/ops_bkup/HRPRD/arch
standby_file_management string
AUTO
SQL> alter system set standby_file_management=MANUAL;

System altered.

SQL> alter database rename file '/u03/oradata/HRPRD/psundo03.dbf' to '/u06/oradata/HRPRD/psundo03.dbf';

Database altered.

SQL> alter system set standby_file_management=AUTO;

System altered.

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL>

ORA-27300: OS system dependent operation:fork failed with status: 11

Got the following error in the alert log:


ORA-27301: OS failure message: Resource temporarily unavailable ORA-27302: failure occurred at: skgpspawn5 ORA-27300: OS system dependent operation:fork failed with status: 11 ORA-27301: OS failure message: Resource temporarily unavailable ORA-27302: failure occurred at: skgpspawn5 ORA-19583: conversation terminated due to error ORA-04030: out of process memory when trying to allocate 1052696 bytes (KSFQ heap,KSFQ Buffers) ORA-27300: OS system dependent operation:fork failed with status: 12 ORA-27301: OS failure message: Not enough space ORA-27302: failure occurred at: skgpspawn3

Solution: The problem was fixed by increasing the PGA from 399M to 500M.

Block change tracking on standby doesn't work

Today I asked how to turn on block tracking changing on physical standby side for Oracle 10g release 2. The Oracle support said it doesn't work even you turn it on.

Here is what they said:"
Block Change Tracking can be enabled at a physical standby database. However, you should note that changed blocks are not recorded, and incremental backups will not be faster during the physical standby as they are receiving and applying redo transactions from the primary database.

When the physical standby becomes a primary database, changes are once again recorded. Following the subsequent Level 0 backup, incremental backups take advantage of change tracking.

Your primary database block change tracking file would not get updated when you perform backup from standby. In 10.2, the CTWR process will only appear when the database is open. So even though you can enable block change tracking at a standby database, which will create the change tracking file, the CTWR process which is responsible for writing to the BCT filewill not start and the standby database will not record changes while the database is in recovery mode. It will start recording changes after the database is opened after a switchover or failover.

Refer Note.262853.1 Ext/Pub 10G RMAN Fast Incremental Backups."

I guess we still have one choice. It is to do the incremental backup at primary side. Since the block tracking changing on primary side will work, incremental backups at primary side may take much less time than the standby side. We can still perform the full backup at standby side as it takes the same amount of time as at primary side.