IDI Operations: Finding the Solution Directory of a Running ibmdisrv Process

Background

I am working with a client that has had ITIM and TDI/IDI installed and running for over a decade, with different groups responsible for the administration of each. They now have around ten ibmdisrv processes running, with some of them configured to run the same Listener ALs (so the one that starts first wins). None of the servers are started with the "-s" flag, so it's a little difficult to figure out their solution directories. 

Each ibmdisrv and ibmditk process needs to have a Solution Directory (aka SOLDIR, SolDir, or current working directory) defined. All default logging is done in the solution directory, and all relative paths used are relative to the solution directory. Basically, the solution directory is very important, and this post will show you how you can find it for a running process.

The solution shown here is for Linux, because that is the most common OS I've seen for running IBM Directory Integrator. The second most popular platform is Windows. If you're running Windows, you can get similar data using the Process Explorer GUI tool.

Solution

The first thing to look at is the process listing for the IDI server java process. The easiest way to find all of these processes running on your server is with this command:

# ps -ef | grep IDILoader

root       5693   5683  0 Jan01 pts/0    00:07:57 /opt/IBM/TDI/ceV10/jvm/jre/bin/java -cp /opt/IBM/TDI/ceV10/IDILoader.jar -Dlog4j2.configurationFile=file:etc/log4j2.xml --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED com.ibm.di.loader.ServerLauncher -d

We see from that output that the PID of this java process is 5693 and the Parent PID (PPID) is 5683. The most straightforward way I've found to find the SolDir for this server is using this command:

# lsof -Pp 5693 | grep cwd

The output should be similar to:

java    5693 root  cwd    DIR              253,0      4096  68731158 /opt/IBM/TDI/prod_V7.1.1

Easy as that, we see that the SolDir for this server is /opt/IBM/TDI/prod_V7.1.1.

Additional Details

We can get more details about the PPID with this command:

# ps -fwwp 5683

UID         PID   PPID  C STIME TTY          TIME CMD

root       5683   5462  0 Jan01 pts/0    00:00:00 /bin/sh /opt/IBM/TDI/ceV10/ibmdisrv -d

What we see here is that the /opt/IBM/TDI/ceV10/ibmdisrv script was called with only the "-d" flag to start this process. When that script runs, it calls the bin/setupCmdLine.sh script, which eventually calls the script bin/defaultSolDir.sh to set the solution directory of the process. (A great post about this and more can be found at this link.) While this is truly great information to know, one drawback is that the solution directory can be overridden with the TDI_SOLDIR environment variable. That's why I like the lsof approach better on Linux. The behavior of lsof on AIX is different and doesn't point to the solution directory like it does on Linux, so this approach may be better there. In either case, it's good to know both.

The Cylance Smart Antivirus agent will ruin your day

I am currently helping a customer move their ITM 6 infrastructure from AIX to Red Hat 8, and the largest hurdle has been the Cylance agent. When doing any kind of enterprise install, my first step is to copy the install files to all of the servers (in this case it is 16 servers: 2 HUB TEMS, 12 RTEMS, 2 TEPS). In its default configuration, the Cylance agent will remove files that it determines are suspicious. In my case, that means that it deleted one or two tar files, and would re-delete them whenever I copied them over again. The cylance log under /opt/cylance/desktop/log showed exactly what it was doing, so we were able to work with the Cylance team to correct this.

After the delete issue was resolved, we found that the Cylance agent was stopping some executables from running, with just a "Segmentation fault" error, and the error still existed after stopping the Cylance agent. This is because even though the agent wasn't running, it has hooks into kernel system calls that leverage a local cache. That took a while to resolve, but we finally got all of the appropriate directories whitelisted.

The last problem encountered was with the Cylance agent's Memory Protection feature. In this case, it caused 'tacmd tepslogin' to fail with a bunch of text to the command line and no information in the normal ITM logs. Looking in the Cylance log file again, I could see that it was blocking some memory action performed by the ITM java executable. That now seems to be resolved.

Hopefully this short post can help others identify these types of issues before throwing their server out the window.

Customizing bash command line completion

What am I talking about?

In the bash shell on Linux, you can type a character or two then hit the TAB key to get a list of the commands that start with those characters. You can do the same to complete the name of a file you're trying to edit or directory you're trying to change to. It turns out that you can customize this command line completion behavior by installing the "bash-completion" package. This package is often installed by default and has been available for several years.

What can you do with bash-completion?

You can have the TAB key complete command arguments for you. For example, the 'curl' command has tons of arguments. You can customize bash to auto-complete the parameters for you. You just need to create a specifically coded file named 'curl' in the /etc/bash_completion.d folder. Here's a great tutorial on creating these command completion scripts:

https://iridakos.com/tutorials/2018/03/01/bash-programmable-completion-tutorial.html

Even more helpful, here is a ton of them that have already been created:

https://github.com/GArik/bash-completion/tree/master/completions

If you've got a command with tons of options, you can use this to make it easier for you or your users to successfully create a working command.

Installing IBM Control Desk 7.6 on RHEL 6.5 in a test environment

The biggest hitch you'll encounter when installing ICD 7.6 on Redhat Linux in a dev/test environment is the error

CTGIN8264E : Hostname failure : System hostname is not fully qualified

And the reason it's a big hitch is because the error is misleading. You do need to have your hostname set to your FQDN, but you also need to have an actual DNS (not just /etc/hosts, but true DNS) A record for your hostname. If you don't already have one that you can update, you can install the package named:

The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server

It is available on the base Redhat install DVD.

Installing IBM Control Desk v7.6 on Red Hat Enterprise Linux 6.6

I just had a difficult time installing ICD 7.6 on RHEL 6.6 and wanted to share one workaround that I used that finally got me around the prerequisite checker failures on Installation Manager.

I've done this successfully in a production environment, but in this case I'm installing this in a VM in my lab that's running under VMWare Workstation, and it's on a NAT network. Every time the Installation Manager runs the prerequisite checker, I kept getting a failure on network.fqdn, even though the hostname was absolutely set to a fully qualified domain name (icd76.mynet.foo). So I started up 'dnsmasq' locally and pointed to my own IP address as my DNS server. That allowed network.fqdn to PASS, but then network.dns would FAIL (!).

Figuring I have my networking configured "good enough" for my small test machine, I simply edited the following files:

PAD_07060000.cfg
PAW_07060000.cfg

in the directory /var/ibm/InstallationManager/bundles/plugins/com.ibm.tivoli.pae.prereq 1.0.1.20157141414/com/ibm/tivoli/pae/prereq/SystemRequirements/unix/ to change:

network.dns=True

to

network.dns=False

And then it worked like a champ!

How did you find those buried files?

If you cd to /var/ibm/InstallationManager/logs and run 'firefox index.xml', you get a great view of all of the Installation Manager log files. In the latest one, the completion message pointed me basically to that directory.

ICO 2.5: Creating a Red Hat 6.5 image with Gnome for use with a VMWare vSphere cloud

Introduction

When creating your private OpenStack-managed vSphere cloud, you're going to need some "images" ("VM Templates" in VMWare terminology) so you can launch/deploy instances. The really sticky part about this configuration is that OpenStack has traditionally only supported the KVM hypervisor, which uses a different disk format than VMWare (KVM uses QCOW2 and VMWare uses VMDK). I found some great QCOW2 images here: http://docs.openstack.org/image-guide/content/ch_obtaining_images.html and some great CentOS VMDK images here: http://osboxes.org. I had some different hurdles with each of those and finally decided just to install RHEL 6.5 from scratch, then modify that VM to work, then create a VM Template that would be automatically discovered by OpenStack as an image. In this post I'll cover the highlights of this technique.

Install Red Hat

Install the "Desktop" option. This will mean that you have to do a little cleanup later, but I'll cover that.

Install VMWare Tools

Go ahead and install VMWare tools just to make your life a little easier.

Configure Yum Repositories and Install Packages

If you're using Red Hat Subscription Manager, you don't have to go through these steps. If you're not using RHSN, you'll need to manually configure some repositories. I found for this exercise that the CentOS repositories, along with the EPEL repository, worked great. To configure the CentOS repositories, create a file named /etc/yum.repos.d/YOU_PICK_A_NAME.repo with the following contents:

[base]

name=CentOS-6 - Base

mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os

#baseurl=http://mirror.centos.org/centos/6/os/x86_64/

gpgcheck=0

#released updates

[updates]

name=CentOS-6 - Updates

mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=updates

#baseurl=http://mirror.centos.org/centos/6/updates/x86_64/

gpgcheck=0

#additional packages that may be useful

[extras]

name=CentOS-6 - Extras

mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=extras

#baseurl=http://mirror.centos.org/centos/6/extras/x86_64/

gpgcheck=0

#additional packages that extend functionality of existing packages

[centosplus]

name=CentOS-6 - Plus

mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=centosplus

#baseurl=http://mirror.centos.org/centos/6/centosplus/x86_64/

gpgcheck=0

enabled=0

#contrib - packages by Centos Users

[contrib]

name=CentOS-6 - Contrib

mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=contrib

#baseurl=http://mirror.centos.org/centos/6/contrib/x86_64/

gpgcheck=0

enabled=0

Then run 'yum repo-list' to verify the repositories were created. Now you can run the following commands:

yum -y epel-release

yum -y cloud-init

yum -y cloud-utils

yum -y heat-cfntools

Configure Networking

Since the "Desktop" option was chosen, the NetworkManager service was installed and enabled. You need to change that with these commands:

service NetworkManager stop

chkconfig NetworkManager off

service network start
chkconfig network

Now run the following commands so that eth0 will get configured properly:

sed -i '/^HWADDR/d' /etc/sysconfig/network-scripts/ifcfg-eth0

echo -n > /etc/udev/rules.d/70-persistent-net.rules

echo -n > /lib/udev/rules.d/75-persistent-net-generator.rules

Configure cloud.cfg

The default configuration will try to contact multiple non-existent IP addresses for metadata, and you don't need that by changing /etc/cloud/cloud.cfg. Add the following line to that file:

datastore_list: [ NoCloud, ConfigDrive, None ]

You *may* also want to set:

disable_root: 0

ssh_pwauth: 1

This will allow you to access the VM as root, and via ssh with a password.

Make a Template

Via the vSphere Web Client or vSphere Windows Client, clone the VM to a template. After you do this, wait a few minutes for OpenStack to "see" this new template as an Image.

Conclusion

Following the above steps should get you a working template from which you can explore the other capabilities of ICO and ICMWO.