Author Archives: Stefan

Release of Audit Vault and Database Firewall 12.1.2 Bundle Patch 7

Today Oracle released the new Bundle Patch for Audit Vault and Database Firewall 12.1.2. The patch can be downloaded as usual on Oracle Metalink as Patchset 21920205 for existing installations. The full installation image for new installations is not yet available on Oracle eDelivery. I guess this will follow in a couple of days. Beside the Bundle Patch, Oracle will also updated the Backup Script to the latest Release. The scripts will be available via My Oracle Support Note 1556200.1

According the readme, the Release 12.1.2 BP7 just contains the October 2015 Patch Set Update for the database. The base platform has been updated with several not precisely specified bug fixes. These include security and stability fixes to Java and the underlying Linux operating system plus the bug fix for the following bug:

Bug Number Description
21395711 ALERT IS RESENT TO SYSLOG WHEN JFWK IS RESTARTED

Since the PSU for October 2015 does includes some critical but fixes for clusterserver (CVSS Rating 10). It is recommended to install this Bundle Patch.

Patch installation

The patch installation is rather simple. Most important is that the following directories have enough free space:

  • 5 GB in /var/lib/oracle
  • 5 GB in /var/tmp
  • 4.5 GB in /root

To install the patch just copy the iso to the AVDF server and run the ruby script. Alternatively you may also mount the iso directly on the Server instead of copy it first. eg. if you run your AVDF in a VM environment. Detailed installation instruction could be taken from the Patch Readme


[root@melete ~]# /bin/mount -oloop,ro /root/avdf-upgrade-12.1.2.7.0.iso /images

[root@melete ~]# yum -c /images/upgrade.repo clean all
Cleaning up Everything

[root@melete ~]# /usr/bin/ruby /images/upgrade.rb
Verifying upgrade preconditions
Mounting boot partition
Removing obsolete files and packages
Applying kernel upgrade
Upgrading system
Remove media and reboot now to fully apply changes.

[root@melete ~]# /sbin/reboot

Broadcast message from root (pts/0) (Mon Nov  9 14:51:46 2015):

The system is going down for reboot NOW!




AVDF Backup

Beside the Bundle Patch, Oracle will also updated the AVDF Backup Script to match the latest Release. The script itself is not yet available, but the new Version will be posted in My Oracle Support Note Audit Vault Server Backup and Restore for Release 12.1.2.5.0 and Prior [1556200.1].

References

Some links related to the Audit Vault and Database Firewall:

WALLET_LOCATION in sqlnet.ora for Container Databases

Recently I’ve setup Oracle Enterprise User Security (EUS) with Oracle Unified Directory (OUD) on my favorite linux test system. Among regular 11.2.0.4 and 12.1.0.2 databases I do also have a 12.1.0.2 Container Database. EUS work like a charm on the regular databases but not on the PDB. 

SQL> conn soe
Enter password: 
ERROR:
ORA-28305: WALLET_LOCATION in sqlnet.ora file for container database is not
supported.


Warning: You are no longer connected to ORACLE.

The error seems to be a bit weird. So fare I’ve explicitly set the wallet location to make sure the wallet it somewhere I decided. I have a shared sqlnet.ora file, where I use $ORACLE_SID in the path for the different instances. An excerpt from my sqlnet.ora file

...
WALLET_LOCATION =
  (SOURCE =
    (METHOD = File)
    (METHOD_DATA = (DIRECTORY = /u00/app/oracle/admin/$ORACLE_SID/wallet)))

ENCRYPTION_WALLET_LOCATION=
 (SOURCE=
  (METHOD=FILE)
   (METHOD_DATA=
    (DIRECTORY=/u00/app/oracle/admin/$ORACLE_SID/tde_wallet/)))
...

The action described for the Oracle Error Message ORA-28305 is clear. Remove WALLET_LOCATION from sqlnet.ora to use EUS also for Container Databases.

SQL> conn soe
Enter password: 
Connected.
SQL> @sousrinf
Database Information
--------------------
- DB_NAME		: TDB12C
- DB_DOMAIN		:
- INSTANCE		: 1
- INSTANCE_NAME 	: TDB12C
- SERVER_HOST		: o-sec
-
Authentification Information
----------------------------
- SESSION_USER		: C##SOE
- PROXY_USER		:
- AUTHENTICATION_METHOD : PASSWORD
- IDENTIFICATION_TYPE	: GLOBAL SHARED
- NETWORK_PROTOCOL	:
- OS_USER		: oracle
- AUTHENTICATED_IDENTITY: SOE
- ENTERPRISE_IDENTITY	: cn=soe,cn=Users,dc=trivadistraining,dc=com
-
Other Information
-----------------
- ISDBA 		: FALSE
- CLIENT_INFO		:
- PROGRAM		: sqlplus@o-sec (TNS V1-V3)
- MODULE		: SQL*Plus
- IP_ADDRESS		:
- SID			: 39
- SERIAL#		: 47117
- SERVER		: DEDICATED
- TERMINAL		: pts/6

PL/SQL procedure successfully completed.

The corresponding Oracle Bug 17758886 has been rejected as “not a Bug”. Oracle® Database Net Services Reference 12c Release 1 (12.1) WALLET_LOCATION does not mention PDB’s. There is only some information in the Oracle® Database Reference 12c Release 1 (12.1) Using LDAP_DIRECTORY_ACCESS with PDBs.

Conclusion

It seems, that with PDB’s it is not possible to explicitly set a wallet location. If the default location is not appropriate for your database environment, you have to use soft links use an alternative location for your wallet.

By the way, the wallet for TDE or for Secure External Password Store (SEPS) is not affected. You may still set WALLET_LOCATION for SEPS or ENCRYPTION_WALLET_LOCATION for TDE.

References

Some links related to this topic.

If time permits, I’ll write a few blog post about setting up and configuring EUS with OUD.

OPatch silent and unattended

In general I use Oracle OPatch interactively in command line mode to install patch set updates. But recently I did patch a system cloud based system, with a confusing network timeout. As expected I did get a broken pipe while executing OPatch. Ok, the system is also damn slow, which is not exactly helpful. Never mind, this was the time to look around for a stable alternative. eg. OPatch should survive potential network / connection lost.

A possible solution would be using screen. Unfortunately screen is not available on the HP-UX system, which I use for this particular Critical Patch Update tests. Therefore I’ve searched in MOS and found two helpful notes about using opatch in silent mode.

First step is to create a response file for OCM to make sure you do not get ask about security updates:

oracle@hpux01:~/ [CPU11204] $cdh/OPatch/ocm/bin/emocmrsp -no_banner -output $cdl/oradba/rsp/ocm_opatch.rsp
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y
The OCM configuration response file (/u00/app/oracle/local/dba/../oradba/rsp/ocm_opatch.rsp) was successfully created.

Second step is to run opatch in silent mode with the response file for the OCM.

oracle@hpux01:~/ [CPU11204] cd /u00/app/oracle/software/ora/CPU_2015_10/21352635/
oracle@hpux01:/u00/app/oracle/software/ora/CPU_2015_10/21352635/ [CPU11204] $cdh/OPatch/opatch apply -silent -ocmrf $cdl/oradba/rsp/ocm_opatch.rsp

Alternatively run opatch apply with nohup. 

oracle@hpux01:~/ [CPU11204] cd /u00/app/oracle/software/ora/CPU_2015_10/21352635/
oracle@hpux01:/u00/app/oracle/software/ora/CPU_2015_10/21352635/ [CPU11204] nohup $cdh/OPatch/opatch apply -silent -ocmrf $cdl/oradba/rsp/ocm_opatch.rsp &

oracle@hpux01:/u00/app/oracle/software/ora/CPU_2015_10/21352635/ [CPU11204] tail -f nohup.out 

Oracle Home       : /u00/app/oracle/product/11.2.0.4
Central Inventory : /u00/app/oraInventory
   from           : /u00/app/oracle/product/11.2.0.4/oraInst.loc
OPatch version    : 11.2.0.3.12
OUI version       : 11.2.0.4.0
Log file location : /u00/app/oracle/product/11.2.0.4/cfgtoollogs/opatch/opatch2015-10-30_08-51-15AM_1.log

Verifying environment and performing prerequisite checks...

At the time of analog and ISDN connections I was used to network interruptions. But today, where one is everywhere online… I guess I must use nohup and screen more often again. 🙂

References

Memory Leak in Network Checksum with new SHA-2 Functions

I’ve just stumbled over an issue with the new checksum algorithm introduced with Oracle 12c. It seams that in certain situation the new SHA-2 function cause a memory leak. A search on My Oracle Support revealed that there is a Bug on AIX. See Bug 19451972 MEMEORY LEAKS WITH SHA512, SHA384, SHA256 ENTRIES IN SQLNET.CRYPTO_CHECKSUM and the corresponding Note 1919000.1 SQLPlus 12c Memory usage Grows (Leaks) While Running Long Query.

Test Case

Nevertheless I have similar issues on a Exadata Machine and my Oracle VM. To verify my issue I’ve used a simple test case, where I start a SQL*Plus script which does:

  1. connect as SCOTT
  2. query some views eg. v$session_connect_info
  3. wait a few seconds
  4. query some views eg. v$session_connect_info
  5. start over with step 1

Since SQL*Plus does not support any loop I just use cat to generate a script with a bunch of connect and SELECT. For this I used the following Template (connect_scott_template.sql):

connect scott/tiger@TDB12A
alter session set nls_date_format='DD.MM.YYYY HH24:MI:SS';
select sysdate from dual;
select sid, osuser, authentication_type, network_service_banner
from v$session_connect_info
where sid=(select sys_context('userenv','sid') from dual);
exec DBMS_LOCK.SLEEP(10);

Based on this template I’ve created my SQL script with a for loop.

for i in {1..720}; do cat connect_scott_template.sql >>connect_scott.sql ; done

If the script runs for a couple of minutes / hour you will see that RSS (real memory size / resident set size) does increase when network integrity check is enabled with SHA512. It remains on the same level for the same test without network integrity check. Below you see the output of my bash session history (with minor optimization for the web 😉 ):

cd /u00/app/oracle/admin/TDB12A/adhoc/nocksum
export TNS_ADMIN=$PWD

nohup sqlplus /nolog @connect_scott.sql &

PID=17185

while [[ $(ps $PID|wc -l) > 1 ]]
do 
echo $(date "+%Y.%m.%d %H:%M:%S $(ps u $PID|tail -1)") >>connect_scott_nocksum_<code>{{EJS16}}</code>.log
sleep 30
done

I’ve started sqlplus and the script with nohup. To collect the rss information I’ve just created a while loop and pipe the output of ps to a log file. For the test with checksum typ SHA512 used an alternative TNS_ADMIN directory with a different sqlnet.ora. My Test did run for about two hours. I’ve but the collected data in an Excel sheet to create the following chart. You see, that both SQL*Plus process require more real memory over time. Nevertheless the required memory for SQL*Plus with SHA512 is definitely higher.

MemoryLeak

Files and References

Below you find the scripts mentioned above as well some MOS references:

Conclusion

It seems that this Bug is a bit more generic than expected. Since the new SHA function would anyway just work for pure 12c environment, it is acceptable to use the old SHA1 Hash until this Bug is fixed.

Oracle CPU / PSU Pre-Release Announcement July 2015

Oracle has published the Pre-Release Announcement for the July Critical Patch Update. This Critical Patch Update contains 193 new security vulnerability fixes across all Oracle products. It looks like that this CPU does contain a bunch of critical security fixes for Oracle databases. Actually there are 10 fixes for security vulnerabilities, 2 of them are remotely exploitable. There is no security fix for client-only installations. Nevertheless the highest CVSS rating is 9.0. I wonder which OS is affected 😉

Beside the high CVSS rating, some core components seems to be affected:

  • Application Express
  • Core RDBMS
  • Java VM
  • Oracle OLAP
  • RDBMS Partitioning
  • RDBMS Scheduler
  • RDBMS Security
  • RDBMS Support Tools

We will see all the details later today, when Oracle is officially releasing the Critical Patch Update for July 2015. Together with my colleagues at Trivadis, we’ll have a closer look and do some testing. See also TVD-Critical
PatchReport™ or TVD-Trivadis eXpert Team Security

More details about the patch will follow soon on the Oracle Security Pages.

DOAG SIG Security Munich 2015

Just finished my presentation about Unified Audit at the DOAG SIG Security in München. It is about Oracle Unified Audit and a few considerations for migrating old standard audit to new policy based unified audit. The slides are available for download  DOAG_SIG_Security_Oracle_Unified_Audit.pdf.

Some impression for the event and my presentation.
DOAG_SIG_SEC_2015_1 DOAG_SIG_SEC_2015_2

Release of Audit Vault and Database Firewall 12.1.2 Bundle Patch 5

Today Oracle released the new Bundle Patch for Audit Vault and Database Firewall 12.1.2. The patch can be downloaded as usual on Oracle Metalink as Patchset 20829881 for existing installations. The full installation image for new installations is not yet available on Oracle eDelivery. I guess this will follow in a couple of days. Beside the Bundle Patch, Oracle also updated the Backup Script to the latest Release.

According the readme, the Release 12.1.2 BP5 contains the April 2015 Patch Set Update for the database as well several bug fix for the base platform. These include security and stability fixes to Java and the underlying Linux operating system plus the bug fix for the following bugs:

Bug Number Description
18730748 THE AUDIT TRAILS CANNOT GET DATA FROM DATABASES CONFIGURED FOR SSL
18081207 PGA_AGGREGATE_LIMIT HIT WHEN QUERYING RECORDS FROM V$UNIFIED_AUDIT_TRAIL
18349496 FOR MSSQL TRACE FILES, COLLECTOR IS NOT COLLECTING THE DATA FROM ACTIVE FILE
20488901 MISSING MAPPING OF CLIENT_IP
17830617 TRANSACTION LOG AUDIT TRAIL CRASHES
20688669 ORA-20105: FAILED TO REMOVE FIREWALL CERTIFICATE FROM ORACLE WALLET

Patch installation

The patch installation is rather simple. Most important is that the following directories have enough free space:

  • 5 GB in /var/lib/oracle
  • 5 GB in /var/tmp
  • 4.5 GB in /root

To install the patch just copy the iso to the AVDF server and run the ruby script. Alternatively you may also mount the iso directly on the Server instead of copy it first. eg. if you run your AVDF in a VM environment. Detailed installation instruction could be taken from the Patch Readme


[root@melete ~]# mount /dev/cdrom /images
mount: block device /dev/cdrom is write-protected, mounting read-only

[root@melete ~]# yum -c /images/upgrade.repo clean all
Cleaning up Everything

[root@melete ~]# /usr/bin/ruby /images/upgrade.rb
Verifying upgrade preconditions
Mounting boot partition
Removing obsolete files and packages
Applying kernel upgrade
Upgrading system
Remove media and reboot now to fully apply changes.

[root@melete ~]# /sbin/reboot

Broadcast message from root (pts/0) (Fri May 15 13:40:50 2015):

The system is going down for reboot NOW!




AVDF Backup

Beside the Bundle Patch, Oracle also updated the AVDF Backup Script to match the latest Release. The script itself did not change. Oracle just added the product version 12.1.2.5.0. See Audit Vault Server Backup and Restore for Release 12.1.2.5.0 and Prior [1556200.1] for more information on the backup script.

References

Some links related to the Audit Vault and Database Firewall:

Oracle CPU / PSU Pre-Release Announcement January 2015

Oracle has published the Pre-Release Announcement for the first Critical Patch Update in 2015. This Critical Patch Update contains 167 new security vulnerability fixes across all Oracle products. It looks like that this CPU does contain a bunch of critical security fixes for Oracle databases. Actually there are 7 fixes for security vulnerabilities, but none of them is remotely exploitable nor are they for client-only installations. Nevertheless the highest CVSS rating is 9.0. I wonder which OS is affected 😉

Beside the high CVSS rating, some core components seems to be affected:

  • Core RDBMS
  • DBMS_UTILITY
  • PL/SQL
  • Recovery
  • Workspace Manager
  • XML Developer’s Kit for C

We will see all the details later today, when Oracle is officially releasing the Critical Patch Update for January 2015. Together with my colleagues at Trivadis, we’ll have a closer look and do some testing. See also Trivadis Critical Patch Updates Report

More details about the patch will follow soon on the Oracle Security Pages.

Oracle Software Appliances and Bash Shellshock

Late September a vulnerability in the bash Shell has been published. The vulnerability also known as shellshock, was classified as extremely critical. Anyway, in the meantime security patch has been released for the different operating systems and bash implementations. A bugfix is also available for Oracle Enterprise Linux, which is used as operating system of the two Oracle software appliances Oracle Audit Vault and Database Firewall and Oracle Key Vault. Oracle has published two My Oracle Support Notes which describe how the patch must be installed on the appliance software. The installation is quite straightforward. Get the patch from the Oracle’s public yum repository and install it on the appliance. 🙂 But be aware, that the two appliance are still runing Oracle Enterprise Linux 5.

Steps to copy, install and verify the bash shell bugfix:

[support@melete ~]$ su -
Password: 

[root@melete ~]# env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
vulnerable
this is a test

[root@melete ~]# rpm -Uvh /tmp/bash-3.2-33.el5_11.4.x86_64.rpm 
warning: /tmp/bash-3.2-33.el5_11.4.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:bash                   ########################################### [100%]

[root@melete ~]# rpm -qa | grep -i bash 
bash-3.2-33.el5_11.4

[root@melete ~]# env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
this is a test

References

Some links related to the bash shellshock vulnerability.

Deploying an Agent from a Windows 12.1.0.4 OMS fails

I’ve just setup a new Oracle Enterprise Manager Cloud Control 12.1.0.4 on Windows Server 2008 R2 for Customer. It’s a small one system setup, which will be used to monitor a few Oracle Databases on Windows and Linux. After the successful setup I’ve tried to deploy agents from the console, using the AgentPull. I have already performed this several times successfully. But somehow it does not work with 12.1.0.4. It fails while creating the plugin archive…

Creation of the plugin archive “e:/oracle/product/gc_inst/em/EMGC_OMS1/sysman/agentpush//plugins/windows_x64/12.1.0.4.0_PluginsOneoffs_233.zip” failed. Check the log file e:/oracle/product/gc_inst/em/EMGC_OMS1/sysman/agentpush/logs/EMAgentPushLogger.log for more details

Solution

It seems, that 12.1.0.4 on windows does have issues creating the agent image when OMS_HOME is on a different drive than SCRATCHPATH.
In my installation the OMS_HOME is on drive D: whereas SCRATCHPATH is on drive C:. The solution is straight forward. I’ve just have to reconfigure the SCRATCHPATH in ssPaths_msplats.properties.

cd $OMS_HOME\oui\prov\resources\
cp ssPaths_msplats.properties ssPaths_msplats.properties.bck

vi ssPaths_msplats.properties

SCRATCHPATH=C:/tmpada

SCRATCHPATH=D:/tmpada

As soon as this has change you may re-deploy the agent from the console.

References

More details to this issue can be found in the MOS note and the related Bug / Patch:

  • EM 12c R4: Installing / Deploying An Agent from a Windows 12.1.0.4 OMS fails when the OMS ORACLE_HOME is not installed on the ‘C:’ drive [1681463.1]
  • GET_AGENTIMAGE SHOULD WORK WHEN OMSHOME IS IN DRIVE DIFFERENT FROM SCRATCHPATH 18938561