Tag Archives: AVDF

Update agent.jar on audit vault server

As I wrote in my post Error installing Audit Vault Agent 12.1.1 on AIX, there is a Audit Vault Agent bug 17058352 on AIX. Unfortunately it hasn’t yet been fixed in the latest bundle patch for Oracle Audit Vault and Database Firewall. If you haven’t changed your default profile in /etc/profile on your AIX server, you will run into the same issues again when trying to update the agents according the patch readme. Rather than downloading and fixing the agent.jar on each AIX system, it is also possible to update the agent.jar on the Audit Vault server before updating and restarting the agents on the monitored servers.

For this just login to the Oracle Audit Vault and Database Firewall server as support and become oracle.

ssh support@melete2

su - root
su - oracle

Locate your agent.jar, backup it and unpack the agentctl

cd /var/lib/oracle/dbfw/av/jlib/
cp agent.jar agent.jar_backup_bugfix_17058352
jar -xf agent.jar bin/agentctl

Update the agentctl and add LOGNAME the the list of pass through variable on line 46.

# Passthrough env vars
# Note: we passthru any vars with "-" invalid character
#
passthru='^TZ$|^LANG$|^LC_|^JAVA_HOME$|^PATH$|^PS1$|^LOGNAME$|-'

Put the updated agentctl script back to the agent.jar and run a regular installation.

jar -uf agent.jar bin/agentctl

You now just can proceed with updating the agent on the AIX servers.

OAV-46599 when trying to add new secure target on AVDV 12.1.1.1

It is the second time that I run into this problem. Therefore, it is time to write a quick note before I struggle a third time. At some point adding a new secure targets no longer works and breaks with an OAV-46599.

OAV 46599

Initially I was a bit confused about the error. Because there haven’t been any changes on the system since the last secure target has been added. But reading the whole error message above gives the correct indication of the root cause. It’s nothing else than an ORA-28001 the password has expired. Lets see which user has an expired password.

SQL> alter session set nls_date_format='DD.MM.YYYY HH24:MI:SS';

Session altered.

SQL> set linesize 160 pagesize 200
SQL> select username,account_status,expiry_date from dba_users where account_status='EXPIRED';

USERNAME                       ACCOUNT_STATUS                   EXPIRY_DATE
------------------------------ -------------------------------- -------------------
ANONYMOUS                      EXPIRED                          17.09.2011 10:21:08
AVREPORTUSER                   EXPIRED                          17.07.2013 21:25:55

It looks like the account AVREPORTUSER is expired. Because I do not know the password yet know whether it was stored somewhere, I’ll just reset the old password. For this I need both password hash’s.

SQL> col name for a15
SQL> col password for a17
SQL> col spare4 for a65
SQL> select name,password,spare4 from user$ where name='AVREPORTUSER';

NAME            PASSWORD          SPARE4
--------------- ----------------- -----------------------------------------------------------------
AVREPORTUSER    F315BBCEBB3F78E7  S:14155D035FEBAB05790EAB47CCC4ACDBD8B728C373EECDABE6EB5FAA9D03

With alter user identified by values I’m able to specify both the 10g and the 11g password hash to reset the password to the same value.

 alter user AVREPORTUSER identified by values 'S:14155D035FEBAB05790EAB47CCC4ACDBD8B728C373EECDABE6EB5FAA9D03;F315BBCEBB3F78E7';

As you can see in DBA_USERS the account has now status open again. Adding secure targets does work again.

SQL> select username,account_status,expiry_date,password_versions from dba_users where username='AVREPORTUSER';

USERNAME                       ACCOUNT_STATUS                   EXPIRY_DATE         PASSWORD
------------------------------ -------------------------------- ------------------- --------
AVREPORTUSER                   OPEN                             11.02.2014 06:39:02 10G 11G

But how has this account become expired? The reason is quite obviously. All AV user do have the Oracle DEFAULT profile which has a limited password life time of 180 days. Therefore, the accounts expire after 180 days. And yes my AVDF test system was set up about 180 days ago. 🙂

SQL> select username,profile from dba_users where username='AVREPORTUSER';

USERNAME                       PROFILE
------------------------------ ------------------------------
AVREPORTUSER                   DEFAULT

SQL> select * from dba_profiles where profile='DEFAULT' and RESOURCE_NAME='PASSWORD_LIFE_TIME';

PROFILE                        RESOURCE_NAME                    RESOURCE LIMIT
------------------------------ -------------------------------- -------- ----------------------------------------
DEFAULT                        PASSWORD_LIFE_TIME               PASSWORD 180

An Oracle Bug has been opened for this issue. BUG 17078860 Relax The profile settings in the AV server’s database

AVDF missing boot partition

While working on the problem with missing RAM on the AVDF test system (see ) I realized, that the linux boot partition is not available by default. 

[root@melete2 log]# ls -al /boot
total 16
drwxr-xr-x  2 root root 4096 Jan 11  2013 .
drwxr-xr-x 24 root root 4096 Jul 11 20:19 ..

[root@melete2 log]# df -kh /boot
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root
                      6.6G  2.2G  4.1G  35% /

Initially I was a bit confused since it contains stuff like grub configuration, inited.img, kernel etc. All stuff that are needed for system boot. Ok, I have not thought about that for the bootloader, the file system does not have to be mounted. From the security point of view it’s even better to not have it mounted. If not mounted nobody can accidentally change something. 😉 Oracle has defined noauto for the boot partition. Therefore the device is not mounted automatically during system boot.

[root@melete2 log]# cat /etc/fstab|grep boot
LABEL=/boot                    /boot                    ext3   noatime,noauto,nodev,nosuid                  1 2

If you need to change the grub configuration just mount the boot partition manually.

[root@melete2 log]# mount /boot

[root@melete2 log]# vi /boot/grub/grub.conf 

[root@melete2 ~]# umount /boot

Audit Vault and Database Firewall 12.1.1 Bundle Patch 1

Oracle just released the new bundle patch for Audit Vault and Database Firewall 12.1.1. The patch can be downloaded on metaling as RPM patch set for existing installations or as full installation images for new installations.

According the readme, the BP1 contains the July 2013 PSU 11.2.0.3.7 for the database as well several bug fix for both the audit vault server and the database firewall.

  • 16993733 Client program column is null when audit collected from Oracle table trail
  • 16699889 Database Vault:Legacy Audit:12c – mapping for a few events missing
  • 16399439 Audit settings UI problem when IE8 browser is used.
  • 16860810 Firewall reports ODF-10001: Internal error: did not find substitution string
  • 15831798 “Print success message checksum content error” seen on login after timeout
  • 16878611 “ATC” files may not be refreshed (file ownership)
  • 16879023 Starting a trail takes a long time – many minutes
  • 16939931 Trails stop when files are deleted

The installation on my test system was quite straightforward. You just have to copy the RPM package on the AV server and start the installation as root with rpm.

[root@melete2 ~]# /bin/rpm -U /tmp/avs-12.1.1.1.0-51_130731.0100.x86_64.rpm
OK
[root@melete2 ~]#

As prerequisite all secure targets and avagents have to be stopped. A simple task on a test environment like I use. But this can become quite cumbersome in a real production environment with a couple of hundred secure targets.

Some MOS links related to this post.

  • Database Firewall 5.x and Oracle Audit Vault and Database Firewall 12.1 bundled patch reference [1328209.1]
  • Patch 16965973 12.1.1.1.0 PS1 bundle patch 1 for Oracle Audit Vault and Database Firewall
  • Patch 16965974 12.1.1.1.0 Full install images for Oracle Audit Vault and Database Firewall
  • Oracle Audit Vault and Database Firewall Readme Release 12.1.1 BP1

AVDF Linux kernel could not recognize whole RAM

After initial setup of an Audit Vault and Database Firewall engineering system, I’ve started to add several audit vault agents and secure targets. In the beginning it went quite smoothly. But after a certain number of secured targets, there were continuously ORA-04031 errors. Most of the errors were related to large pool and PX Msg buffers issues. The analysis of the trace files has shown interesting stuff. 😉 But more on that in a later blog post. The real problem is the available memory.

Symptoms

The Audit Vault and Database Firewall engineering system is running on a HP ProLiant BL465c Gen 8. It comes with 32GB Memory. Should actually be sufficient for a system engineering. It turned out that the 32GB are not recognized by operating system. As you can see below the system has just 3GB memory in total.

[root@melete2 ~]# free
                     total    used   free shared buffers  cached
Mem:               3048108 2385888 662220      0   10720 1525036
-/+ buffers/cache:  850132 2197976
Swap:              4194296  453564 3740732

Reviewing dmesg shows that we lose 29 GB of memory.

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-300.39.5.el5uek (mockbuild@ca-build56.us.oracle.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Wed Mar 13 11:26:53 PDT 2013
Command line: ro root=/dev/vg_root/lv_root console=tty9 udevtimeout=10
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000bddde000 (usable)
 BIOS-e820: 00000000bddde000 - 00000000bde0e000 (ACPI data)
 BIOS-e820: 00000000bde0e000 - 00000000d0000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 000000083efff000 (usable)
DMI 2.7 present.
last_pfn = 0x83efff max_arch_pfn = 0x400000000
MTRR default type: uncachable
MTRR fixed ranges enabled:
  00000-9FFFF write-back
  A0000-BFFFF uncachable
  C0000-FFFFF write-back
MTRR variable ranges enabled:
  0 base 000000000000 mask FFFF80000000 write-back
  1 base 000080000000 mask FFFFC0000000 write-back
  2 disabled
  3 disabled
  4 disabled
  5 disabled
  6 disabled
  7 disabled
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
e820 update range: 00000000c0000000 - 000000083efff000 (usable) ==> (reserved)
WARNING: BIOS bug: CPU MTRRs don't cover all of memory, losing 29679MB of RAM.
------------[ cut here ]------------

Cause

According to an Oracle Metalink Note 1448147.1 this problem is related to a BIOS issue.

Solutions and Workaround

The solution described in Oracle Metalink Note 1448147.1 is to upgrade the BIOS or disable MTRR in kernel. Since BIOS upgrade is not an option for this environment I’ll try to workaround by disable MTRR.

Disable MTRR

Changing the grub.conf is basically quite easy if you find the boot files. When I first try it, I’d realized that there is no grub configuration available. It seems that Oracle decided to not mount /boot at startup. So it is mandatory to first mount the boot partition. Afterward you just can add disable_mtrr_trim as additional kernel option. 

[root@melete2 ~]# mount /boot

[root@melete2 ~]# df -kh /boot
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             145M   26M  112M  19% /boot

[root@melete2 ~]# vi /boot/grub/grub.conf 
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/vg_root/lv_root
#          initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Audit Vault Server 12.1.1.0.0
        root (hd0,0)
        kernel /vmlinuz-2.6.32-300.39.5.el5uek ro root=/dev/vg_root/lv_root console=tty9 
udevtimeout=10 disable_mtrr_trim
        initrd /initrd-2.6.32-300.39.5.el5uek.img
title Audit Vault Server 12.1.1.0.0
        root (hd0,0)
        kernel /vmlinuz-2.6.32-300.38.1.el5uek ro root=/dev/vg_root/lv_root console=tty9 
udevtimeout=10 disable_mtrr_trim
        initrd /initrd-2.6.32-300.38.1.el5uek.img

[root@melete2 ~]# reboot

Broadcast message from root (pts/0) (Thu Jul 11 20:17:56 2013):

The system is going down for reboot NOW!
[root@melete2 ~]# Connection to melete2 closed by remote host.
Connection to melete2 closed.

After reboot we now have 32GB memory available.

[root@melete2 ~]# free
                      total     used     free shared buffers  cached
Mem:               33024372  3930724 29093648      0   17868 2640744
-/+ buffers/cache:  1272112 31752260
Swap:              14680056        0 14680056

Unfortunately, the configuration of the AVDF appliance is not automatically updated to use the extra memory. We have to do some manual changes.

Update Kernel Parameters

The kernel setting have to be changed to allow a bigger SGA. See Metalink Note 1529433.1 for more detailed information on how calculate and set the kernel parameters. For the engineering system we will define a SGA with 20GB therefor we set the shmmax and shmall as follows:

[root@melete2 ~]# vi /etc/sysctl.conf
…
kernel.shmmax=23622320128
kernel.shmall=5368709120
...
[root@melete2 ~]# sysctl -p

Increase SWAP

With 32GB memory, it is also advisable to enlarge the swap space. I’ve discussed this already in the blog post Resize swap space on linux. Since the AVDF appliance does use logical volumes it’s even a bit easier.

[root@melete2 ~]# swapoff -v /dev/vg_root/lv_swap

[root@melete2 ~]# lvresize /dev/vg_root/lv_swap -L +8G

[root@melete2 ~]# mkswap /dev/vg_root/lv_swap

[root@melete2 ~]# swapon -v /dev/vg_root/lv_swap

Increase SGA

Finally we can increase the SGA.


SQL> alter system set sga_max_size=20G scope=spfile;
System altered.

SQL> alter system set sga_target=20G scope=spfile;
System altered.

SQL> startup force

Conclusion

Although AVDF is an appliance, it is mandatory to examine the system after installation. Eg. are there errors in the log files in /var/log, memory, storage etc. available. The solution described here makes it possible to use all the memory. Nevertheless, the appliance has been adjusted to an extent where is necessary to consider whether the support is still archive. If you run into a similar issue on your production AVDF setup I would recommend opening an Oracle SR. Looking forward to the next AVDF patchset. I hope this system stays patchable.

References

Some links related to this post.

  • Linux kernel could not recognize whole RAM [1448147.1]
  • Upon startup of Linux database get ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device[301830.1]
  • Requirements for Installing Oracle Database 12.1 on RHEL5 or OL5 64-bit (x86-64) [1529433.1]
  • Requirements for Installing Oracle 11gR2 RDBMS on RHEL (and OEL) 5 on AMD64/EM64T [880989.1]
  • Master Note of Linux OS Requirements for Database Server [851598.1]

Error installing Audit Vault Agent 12.1.1 on AIX

The Problem

During the setup of the current audit vault agent 12.1.1 on AIX, I’ve run into issues. Depending on the configuration of the AIX environment, the agent can not be installed at all.

avagent@host:/u00/app/avagent/ [avagent] java -jar agent.jar -d /u00/app/avagent/product/avagent
/u00/app/avagent/product/avagent/bin/agentctl[56]: LOGNAME: is read only
Error while executing command: [sh, /u00/app/avagent/product/avagent/bin/agentctl, fixperms]
avagent@host:/u00/app/avagent/ [avagent]

The problem is in the for loop on line 56 of agentctl where it tries to unset environment variables. Specifically, the environment variable LOGNAME can not be reset. On our AIX LOGNAME has been defined as read only in /etc/profile.

# Unset all env vars
#
for var in <code>{{EJS26}}</code>; do
  $ECHO $var | $EGREP "$passthru" > /dev/null

  # If no match, i.e. not a passthru then unset
  if [ $? -eq 1 ]; then
    unset $var
  fi
done

The Solutions

Change OS default profile

One solution would be to change the default profile on the OS. For this just open /etc/profile and comment out line 37. But I assume for most of us it is not an option to change the default profile.

# System wide profile.  All variables set here may be overridden by
# a user's personal .profile file in their $HOME directory.  However,
# all commands here will be executed at login regardless.

trap "" 1 2 3 
#readonly LOGNAME

Change the audit agent

The alternate solution is to update the agent.jar and fix agentctl. Get the current agent.jar from the audit vault server and extract the agentclt script.

jar -xf agent.jar bin/agentctl

Update the agentctl and add LOGNAME the the list of pass through variable on line 46.

# Passthrough env vars
# Note: we passthru any vars with "-" invalid character
#
passthru='^TZ$|^LANG$|^LC_|^JAVA_HOME$|^PATH$|^PS1$|^LOGNAME$|-'

Put the updated agentctl script back to the agent.jar and run a regular installation.

jar -uf agent.jar bin/agentctl

The Bugfix

The problem was reported to Oracle and can be tracked using the bug number 17058352.

By the way if you’re using multiline shell prompts agentctl will fail on the same code on any OS. Here you may simple workaround by setting a single line prompt.

New Oracle Audit Vault and Database Firewall

In the hustle and bustle of the Christmas season, it went under that Oracle had released a new version of Oracle Audit Vault respectively Oracle Audit Vault and Database Firewall. This weekend I found some time to take a first look into the new release.

What’s New

About a year ago Oracle released the Audit Vault Server 10.3. (see New release of Oracle Audit Vault). During this update Oracle mainly moved internally to a 11.2.0.3 database. The architecture has remained more or less the same. But this has changed now. Oracle is trying to complete its security portfolio. Therefore Oracle has merged the two Oracle Audit Vault and Oracle Database Firewall into the new Oracle Audit Vault and Database Firewall. From the security officer point of view it is definitely more interesting to only have one platform. On the other hand a software appliance is one of the favorites of the DBA and Unix admins. What about, updates, HA, backup & recovery etc? I’ll try to consider these thoughts in a later post on installing and configuring the new Oracle Audit Vault and Database Firewall.

Some short notes on the new features:

  • Oracle Audit Vault and Database Firewall is released as a software appliance-based platform
  • Internally Oracle does use Oracle 11.2.0.3 including Advance Security and Database Vault to enforce Database security and segregation of duties
  • One simple setup does install and configure the operating system, software, database, web frontend etc
  • Audit Vault Agents for:
  • Oracle Database 10g
  • Oracle Database 11g
  • Microsoft SQL Server 2000
  • Microsoft SQL Server 2005
  • Microsoft SQL Server 2008
  • Sybase Adaptive Server Enterprise (ASE) versions 12.5.4 to 15.0.x
  • IBM DB2 version 9.x (Linux, UNIX, Microsoft Windows)
  • Solaris operating system
  • Oracle ACFS
  • Microsoft Windows Server 2008
  • Microsoft Windows Server 2008 R2
  • Microsoft Active Directory 2008
  • Microsoft Active Directory 2008 R2 on 64 bit

New Architecture

As initially mentioned Audit Vault and Database Firewall are moving closer. Oracle Audit Vault is now also the data storage and analysis platform for the Oracle Database Firewall. Former Database Firewall Management Server is eliminated and thus is replaced with Oracle Audit Vault.

OverviewAVDF

An important note here is that Oracle Audit Vault can not be installed on different platforms as before. It is rather a software appliance like the Oracle Database Firewall. The license for each Oracle Audit Vault and Oracle Database Firewall includes always a license for Oracle Enterprise Linux as well. To install only the appropriate hardware is required. This can be a virtual or a physical host. To setup my test environment, I’ve use as usual virtual servers.

Oracle AVDF Requirements

To install Oracle AVDF the following minimal Hardware Requirements must be met. See as the online installation guide for more details on the installation requirements in particular for the supported secured target products (agents).

  • x86 64-bit Server
  • 2 GB Ram
  • single hard drive 125 GB
  • 1 NIC for Audit Vault Server
  • 1 NIC for Database Firewall Proxy Mode
  • 2 NICs for Database Firewall DAM Mode (monitoring)
  • 3 NICs for Database Firewall DPE Mode (blocking)

In addition to the hardware the following software is required to begin the installation:

  • Oracle Linux Release 5 Update 8 for x86_64 (64 Bit) V31120-01 (3.7GB)
  • Oracle Audit Vault and Database Firewall (12.1.0.0.0) – Server V35715-01 (3.4GB)
  • Oracle Audit Vault and Database Firewall (12.1.0.0.0) – Database Firewall V35716-01 (3.1GB)

The server can not be used for other activities, setup of either Oracle Audit Vault or Oracle Database Firewall will completely reimage the server. But I’ll post more details on the installation later this month.

Resources

Links all around the new Oracle Audit Vault and Database Firewall…