RMAN-06136: ORACLE error from auxiliary database: ORA-19563: header validation failed for file

I have an script to refresh databases every night. This morning, it failed with error


RMAN-06136: ORACLE error from auxiliary database: ORA-19563: header validation failed for file .

No scripts modification recently. By a little research, it's been found out that another DBA added a data file with the same name on the production database. The file been added didn't cause any problem in the production database since it's added to a different mounting point. But the when refreshing, it maps to the same mounting point. As a result, the file got overwritten.

The solution is obviously to change the name at the source database. But since it's production, we can not do it without asking down time. So the work around is to add the following line to the init file of the database to be refreshed.

'/u09/oradata/HCMPROD/hrapp2.dbf','/u15/oradata/hcmstg/hrapp100.dbf',

Grid Control issue to configure a database in a different domain

Grid Control version: 10.2.0.5.

The repository domain is dc.gov but the database in question is located at in.dc.gov. When configuring the database, got the following error:

oracle.sysman.emSDK.emd.comm.CommException: java.net.SocketException: Connection timed out:could be due to invalid address

It's resolved after SA did something for the network. Here's what's been done.


search in.dc.gov dc.gov
nameserver 10.201.1.241
nameserver 10.201.1.242

Looks like the existing nameservers are not good enough.

Z/Linux Oracle 10gR2 installation reference

There's a document about recommended articles on database installation on Z/Linux.
Recommended metalink articles for Oracle Database 10gR2 installation on zLinux (s390x) [ID 761261.1]

Here are some of the recommendations:

General: Note 415182.1 - DB Install Requirements Quick Reference - zSeries based Linux
Note 851598.1 - Linux x86, x86-64, and s390x Requirements Reference List
Red Hat Enterprise Linux (RHEL):
Note 376183.1 - Defining a "default RPMs" installation of the RHEL OS
Note 420382.1 - Requirements for Installing Oracle 10gR2 RDBMS on RHEL 4 on zLinux (s390x)
Suse Linux Enterprise Server (SLES):
Note 386391.1 - Defining a "default RPMs" installation of the SLES OS
Note 431443.1 - Requirements for Installing Oracle 10gR2 RDBMS on SLES 9 zLinux (s390x)Note 471165.1 - Additional steps to install 10gR2 RAC on IBM zSeries Based Linux (SLES10)Note 556906.1 - Requirements for Installing Oracle 10gR2 RDBMS on SLES 10 zLinux (s390x)

References

NOTE:851598.1 - Linux OS Requirements Reference List for Database Server
NOTE:415182.1 - DB Install Requirements Quick Reference - zSeries based Linux
NOTE:420382.1 - Requirements for Installing Oracle 10gR2 RDBMS on RHEL 4 on zLinux (s390x)
NOTE:431443.1 - Requirements for Installing Oracle 10gR2 RDBMS on SLES 9 zLinux (s390x)NOTE:471165.1 - Additional steps to install 10gR2 RAC on IBM zSeries Based Linux (SLES10)NOTE:556906.1 - Requirements for Installing Oracle 10gR2 RDBMS on SLES 10 zLinux (s390x)
NOTE:741646.1 - Requirements for Installing Oracle 10gR2 RDBMS on RHEL 5 on zLinux (s390x)

Ensure you have prerequisite rpms to install Oracle Server 10gR2 on Linux on IBM zSeries [ID 1086769.1]

Requirements for Installing Oracle 10gR2 RDBMS on RHEL 5 on zLinux (s390x) [ID 741646.1]

Can not start dbca and sqlplus after 10gR2 installation on Z/Linux

After installed Oracle 10g and patched it to 10.2.0.4. I'm trying to create a database and getting the following errors:

[oracle@zLinux70 ~]$ dbcaWarning: JIT compiler "jitc" not found. Will use interpreter.UnsatisfiedLinkError exception loading native library: njni10Exception in thread "main" java.lang.UnsatisfiedLinkError: get at oracle.net.config.Config.getNetDir(Unknown Source) at oracle.net.config.Config.initConfig(Unknown Source) at oracle.net.config.Config.(Unknown Source) at oracle.sysman.assistants.util.NetworkUtils.(NetworkUtils.java:225) at oracle.sysman.assistants.util.step.StepContext.(StepContext.java:264) at oracle.sysman.assistants.dbca.backend.Host.(Host.java:686) at oracle.sysman.assistants.dbca.ui.UIHost.(UIHost.java:205) at oracle.sysman.assistants.dbca.ui.InteractiveHost.(InteractiveHost.java:54) at oracle.sysman.assistants.dbca.Dbca.getHost(Dbca.java:160) at oracle.sysman.assistants.dbca.Dbca.execute(Dbca.java:94) at oracle.sysman.assistants.dbca.Dbca.main(Dbca.java:180)

Then I tried sqlplus and also it's not working.
[oracle@zLinux70 ~]$ sqlplus /nologsqlplus: error while loading shared libraries: /u01/app/oracle/oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1: cannot restore segment prot after reloc: Permission denied

By searching on Oracle support, I found the document
DBCA fails: UnsatisfiedLinkError exception loading native library: njni10 in z/Linux [ID 875013.1]. It says to disable SELinux.

Solution
Modify /etc/selinux/config file to set SELINUX=disabled as follows:

SELINUX=disabled.
SELINUXTYPE=targeted

The Linux admin had to bounce the server after the change.

[oracle@zLinux70 ~]$ /usr/sbin/sestatusSELinux status: disabled
[oracle@zLinux70 ~]$ /usr/sbin/getenforceDisabled

The issue is resolved after the bounce.

Oracle 10gR2 installation on Z/Linux 5

I'm installing Oracle 10gR2 on Z/Linux 5.

While starting the installer, got the following errors:
JVMDG080: Cannot find class com/ibm/jvm/Trace
JVMXM012: Error occurred in final diagnostics initialization
Could not create the Java virtual machine.

Did export JAVA_COMPILER=NONE according to document 400785.1 then started it again. Now got a different error:

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-07-26_04-13-20PM. Please wait ...[oracle@zLinux70 database]$ The java class could not be loaded. java.lang.ExceptionInInitializerError

Then I tried the flag -ignoreSysPrereqs, and got the following errors:
Exception java.lang.ExceptionInInitializerError occurred..java.lang.ExceptionInInitializerError at java.awt.Component.(Component.java:548) at oracle.sysman.oii.oiif.oiifm.OiifmGraphicInterfaceManager.(OiifmGraphicInterfaceManager.java:222) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.createInterfaceManager(OiicSessionInterfaceManager.java:194) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.getInterfaceManager(OiicSessionInterfaceManager.java:203) at oracle.sysman.oii.oiic.OiicInstaller.getInterfaceManager(OiicInstaller.java:437) at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:932) at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:872)Caused by: java.lang.NullPointerException at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2171) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2006) at java.lang.Runtime.loadLibrary0(Runtime.java:824) at java.lang.System.loadLibrary(System.java:910) at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:76) at java.security.AccessController.doPrivileged1(Native Method) at java.security.AccessController.doPrivileged(AccessController.java:287) at sun.awt.NativeLibLoader.loadLibraries(NativeLibLoader.java:81) at sun.awt.DebugHelper.(DebugHelper.java:56) ... 7 moreException in thread "main" java.lang.NoClassDefFoundError: oracle/sysman/oii/oiif/oiifm/OiifmMainFrame at oracle.sysman.oii.oiif.oiifm.OiifmGraphicInterfaceManager.(OiifmGraphicInterfaceManager.java:222) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.createInterfaceManager(OiicSessionInterfaceManager.java:194) at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.getInterfaceManager(OiicSessionInterfaceManager.java:203) at oracle.sysman.oii.oiif.oiifm.OiifmAlert.(OiifmAlert.java:151) at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:990) at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:872)

It's working after run the verification utility supplied by Oracle. It's documented as
Ensure you have prerequisite rpms to install Oracle Server 10gR2 on Linux on IBM zSeries [ID 1086769.1]

[ServerOps1@zLinux70 software]$ sudo rpm -Uvh ora-val-rpm-EL5-DB-10.2.0.4-1.s390x.rpm
Preparing... ########################################### [100%]
1:ora-val-rpm-EL5-DB ########################################### [100%]
*************************************************************************************
* Validation complete - please install any missing packages *
* Note that these packages cannot be fully validated automatically: *
* glibc-dev *
* libaio *
* Please check the output of the following command *
* rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep glibc-dev *
* rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep libaio *
* which should return output similar to the following: - *
* *
* glibc-devel-2.5-24 (s390x) *
* glibc-devel-2.5-24 (s390) *
* *
* libaio-0.3.106-3.2 (s390x) *
* libaio-0.3.106-3.2 (s390) *
*************************************************************************************
glibc-devel-2.5-49.el5_5.4 (s390x)
glibc-devel-2.5-49.el5_5.4 (s390)
libaio-devel-0.3.106-3.2 (s390)
libaio-0.3.106-3.2 (s390x)
libaio-0.3.106-3.2 (s390)
libaio-devel-0.3.106-3.2 (s390x)
[ServerOps1@zLinux70 software]$ rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep glibc-dev
glibc-devel-2.5-49.el5_5.4 (s390x)
glibc-devel-2.5-49.el5_5.4 (s390)
[ServerOps1@zLinux70 software]$ rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"grep libaio
libaio-devel-0.3.106-3.2 (s390)
libaio-0.3.106-3.2 (s390x)
libaio-0.3.106-3.2 (s390)
libaio-devel-0.3.106-3.2 (s390x)
[ServerOps1@zLinux70 software]$

Bug : ORA-19921: MAXIMUM NUMBER OF 128 ROWS EXCEEDED CONNECTING TO TARGET

While connecting to target database via RMAN, got the following errors:

RMAN-6900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT rowRMAN-6901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rowsORACLE error from target database:ORA-19921: maximum number of 128 rows exceeded.

By searching, there's an Oracle bug filed. Bug number is Bug 8461724.

But the official Oracle document of the bug says:

Workaround: no.
It's fixed in 11g release 2.

Upgrading a database from 10g to 11g is a project takes months because testing has to be performed before production upgrade. Here is how it's resolved. It might not apply to your case.

SQL> select sid, serial#, program from v$session where program like '%rman%';
SID SERIAL# PROGRAM---------- ---------- ------------------------------------------------ 272 10130 rman@asmpu33 (TNS V1-V3) 353 20880 rman@asmpu33 (TNS V1-V3)
SQL> alter system kill session '272,10130';
System altered.
SQL> alter system kill session '353,20880';
System altered.


By killing the two rman sessions, the issue is resolved.