IBM Identity Manger recons hanging and apparent memory leak in JVM

Background

Two-node WebSphere cell, with two clusters - ITIM_Application and ITIM_Messaging, so two application servers per node (timapp1 and timmsg1, timapp2 and timmsg2). It's a fairly large system (6ooo services and 200000 person objects), and it has been running for almost a decade. It started as ISIM 6.0 and is now at version 10.0. The problem we initially saw was that scheduled recons would just go into a Pending state, and others wouldn't be started at all. The log files on both servers looked very similar, and nothing really stuck out, other than errors about hung threads. After more digging, I saw that timapp2, on the secondary server, would start and almost immediately start growing in heap memory size (I could see the memory usage in 'top' and also with 'jconsole'). Also, that application server would have very high CPU usage (200-500% on  a 6-cpu VM). The workaournd was to restart all of the VMs once things got into a hang condition.

Solution

In poking around, I found that the Service Integration Bus had almost a million messages and some of those were several YEARS old. I could see this from the WebSphere Admin Console, navigating to Service Integration->Buses->itim_bus->Messaging engines->ITIM_Application.000-itim_bus->Queue points. This showed me the queue depth of all queues. I then clicked on the one with the greatest depth (almost 500,000 messages), then clicked on the Runtime tab->Messages-> (first message). That showed me the timestamp of when the message was placed on the queue.

So I waited for a time when the system was quiet, then stopped WebSphere entirely on both nodes, then cleared the SIB following these instructions: https://www.ibm.com/docs/en/sig-and-i/10.0.2?topic=system-clearing-service-integration-bus. Then I started WebSphere on the primary node (the dmgr node), then the second node. With the SIB cleared out, the performance of both Application app servers was normal, with no excessive heap memory usage.

COMPLETELY cryptographically synchronizing two IBM Directory Servers

Background

This documentation exists to cryptographically sync two IBM Directory Server instances. That mostly works for the main LDAP directory data - essentially anything under the visible suffixes. HOWEVER, if you need to also cryptographically synchronize the data in the invisible "cn=Configuration" suffix, that's what this post is for.

The story of how I came across this issue is a normal one for a consultant. I was hired to help a customer work on an decade-old ISIM implementation that had been customized and configured by several different implementers. The customer needed to sunset their old AIX LDAP server and move the functionality to a new Red Hat Linux server. The existing server had multiple users defined with names like "cn=acmereadonly" or "cn=xyzservice", where there are multiple (but unknown and undocumented) integrations using these credentials, but no one actually knew the passwords. These users were all visible in ibmslapd.conf, with their AESS256-encrypted passwords, so it was extremely important to copy these users and their encrypted passwords to the new server so that existing integrations would continue to work.

There are actually TWO .KSF files

There are actually TWO .ksf files for an LDAP instance:

ibmslapddir.ksf - This one is used in the steps in the above link. It contains the encryption key and salt for encryption of data in the LDAP DIT. 

ibmslapdcfg.ksf - This one isn't mentioned at all in the the link above, and it is used for encryption of data in ibmslapd.conf, which includes the "hidden" cn=Configuration suffix.

Both of the above files are needed to accomplish my goal. 

What are the steps?

I'll start with the steps from the original link, clean them up a little and add the additional ones needed. 

To cryptographically synchronize two server instances, assuming that you created the first server instance do the following steps.

1. Create the second server instance, but do not start the server instance. do not run the idsbulkload command, or run the idsldif2db command on the second server instance.
2. Copy the ibmslapddir.ksf file (the ldap key stash file) AND the ibmslapdcfg.ksf file from the first server instance to the second server instance. The file is in the idsslapd-instance_name\etc directory on Windows systems, or in the idsslapd-instance_name/etc directory on AIX®, Linux, and Solaris systems. The instance_name is the name of the server instance.
3. Replace the ibmslapddir.ksf file AND the ibmslapdcfg.ksf file of the target server instance with the ibmslapddir.ksf file AND the ibmslapdcfg.ksf file of the source server instance. 
4. Edit the ibmslapd.conf file on the second server to set the value of ibm-slapdCryptoSync: under dn: cn=Configuration to be the exact same as the value on the first server.
5. If you have any users with passwords defined in ibmslapd.conf on the first server, you can copy those entries directly into the same place in the ibmslapd.conf file on the second server. They will be correctly loaded when the server starts.
6. Run the idsbulkload command from the second server instance.
7. Start the second server instance.
8. If you copied any users in step 5 above, try to log in with them on the second server instance, and you should be successful.