IBM Identity Manger recons hanging and apparent memory leak in JVM

Background

Two-node WebSphere cell, with two clusters - ITIM_Application and ITIM_Messaging, so two application servers per node (timapp1 and timmsg1, timapp2 and timmsg2). It's a fairly large system (6ooo services and 200000 person objects), and it has been running for almost a decade. It started as ISIM 6.0 and is now at version 10.0. The problem we initially saw was that scheduled recons would just go into a Pending state, and others wouldn't be started at all. The log files on both servers looked very similar, and nothing really stuck out, other than errors about hung threads. After more digging, I saw that timapp2, on the secondary server, would start and almost immediately start growing in heap memory size (I could see the memory usage in 'top' and also with 'jconsole'). Also, that application server would have very high CPU usage (200-500% on  a 6-cpu VM). The workaournd was to restart all of the VMs once things got into a hang condition.

Solution

In poking around, I found that the Service Integration Bus had almost a million messages and some of those were several YEARS old. I could see this from the WebSphere Admin Console, navigating to Service Integration->Buses->itim_bus->Messaging engines->ITIM_Application.000-itim_bus->Queue points. This showed me the queue depth of all queues. I then clicked on the one with the greatest depth (almost 500,000 messages), then clicked on the Runtime tab->Messages-> (first message). That showed me the timestamp of when the message was placed on the queue.

So I waited for a time when the system was quiet, then stopped WebSphere entirely on both nodes, then cleared the SIB following these instructions: https://www.ibm.com/docs/en/sig-and-i/10.0.2?topic=system-clearing-service-integration-bus. Then I started WebSphere on the primary node (the dmgr node), then the second node. With the SIB cleared out, the performance of both Application app servers was normal, with no excessive heap memory usage.

IBM Identity Manger: Adding a new attribute

Background

The documentation on this is lacking on this topic, and the example LDIF file provided is in a format that will cause idsldapmodify to fail with a painfully obtuse error message. So I'm documenting this here to help the next person avoid some headache.

Details

Here's the documentation on how to add an attribute. In there, it provides this LDIF file as an example:

dn: cn=schema changetype: modify add: attributetypes attributetypes: ( myAttribute-oid NAME ( 'myAttribute' ) DESC 'An attribute I defined for my LDAP application' EQUALITY 2.5.13.2 SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 {200} USAGE userApplications ) - add: ibmattributetypes ibmattributetypes: ( myAttribute-oidDBNAME ( 'myAttrTable' 'myAttrColumn' ) ACCESS-CLASS normal LENGTH 200 )

However, if you try to import that LDIF file with the idsldapmodify command given, you'll get the following error:

ldapmodify: invalid format (line 5 of entry: cn=schema)

After a bit of pain, I figured out that the whitespace in the lines are the problem. You need to either concatenate the continued lines together OR have some white space at the front of the continued lines. Additionally, there should be a space in betwee "oid" and "DBNAME". So if you change the file to the following, you will no longer get the error:

dn: cn=schema
changetype: modify
add: attributetypes
attributetypes: ( myAttribute-oid NAME ( 'myAttribute' ) DESC 'An attribute I defined for my LDAP application'    EQUALITY 2.5.13.2 SYNTAX 1.3.6.1.4.1.1466.115.121.1.15    {200} USAGE userApplications )
-
add: ibmattributetypes
ibmattributetypes: ( myAttribute-oid DBNAME ( 'myAttrTable' 'myAttrColumn' ) ACCESS-CLASS normal LENGTH 200 )
What about updating the objectclass?
That's what I was wondering! Simply adding an attribute doesn't really help you much until you add it as a MUST or a MAY attribute to an objectclass. IBM has an almost complete guide to doing that here, But it is incomplete. For one thing, the format of the data is wrong again. The long attribute values should all be on a single line, like this:
dn: cn=schema
changetype: modify
replace: objectclasses
objectclasses: ( 2.16.840.1.113730.3.2.2 NAME 'inetOrgPerson' DESC 'Defines entries representing people in an organizations enterprise network.' SUP 'organizationalPerson' Structural MAY ( audio $ businessCategory $ carLicense $ departmentNumber $ employeeNumber $ employeeType $ givenName $ homePhone $ homePostalAddress $ initials $ jpegPhoto $ labeledURI $ mail $ manager $ mobile $ pager $ photo $ preferredLanguage $ roomNumber $ secretary $ uid $ userCertificate $ userSMIMECertificate $ x500UniqueIdentifier $ mytest ) )
Additionally, where did that value for "objectclasses" come from?? Well, that came from this command:
ldapsearch -b cn=schema -s base objectclass=* objectclasses | grep \'inetOrgPerson\'
Running that, you will get a result similar to:
objectclasses=( 2.16.840.1.113730.3.2.2 NAME 'inetOrgPerson' DESC 'Defines entries representing people in an organizations enterprise network.' SUP organizationalPerson STRUCTURAL MAY ( audio $ businessCategory $ carLicense $ departmentNumber $ displayName $ employeeNumber $ employeeType $ givenName $ homePhone $ homePostalAddress $ initials $ jpegPhoto $ labeledURI $ mail $ manager $ mobile $ o $ pager $ photo $ preferredLanguage $ roomNumber $ secretary $ uid $ userCertificate $ userPKCS12 $ userSMIMECertificate $ x500UniqueIdentifier ) )
Modify that to include your new attribute, then put that into the LDIF file. Then run idsldapmodify to import that LDIF file into your server. After that, you'll have your new attribute added to the inetOrgPerson objectclass.