Using vi to edit files on an AWS instance can cause strange characters to be inserted

 This is a short one because I came across it the other day and I almost freaked out. I ssh'd into an AWS Alpine Linux instance using MobaXterm on Windows, and I edited a file with vi. I then saved it and re-opened it, and any lines that I added or modified were just messed up. I didn't get to the root of the issue, but I found that using the nano editor did NOT have this same problem. Strange, but wanted to document it in the hope that it may help someone else.

IBM Identity Manger recons hanging and apparent memory leak in JVM

Background

Two-node WebSphere cell, with two clusters - ITIM_Application and ITIM_Messaging, so two application servers per node (timapp1 and timmsg1, timapp2 and timmsg2). It's a fairly large system (6ooo services and 200000 person objects), and it has been running for almost a decade. It started as ISIM 6.0 and is now at version 10.0. The problem we initially saw was that scheduled recons would just go into a Pending state, and others wouldn't be started at all. The log files on both servers looked very similar, and nothing really stuck out, other than errors about hung threads. After more digging, I saw that timapp2, on the secondary server, would start and almost immediately start growing in heap memory size (I could see the memory usage in 'top' and also with 'jconsole'). Also, that application server would have very high CPU usage (200-500% on  a 6-cpu VM). The workaournd was to restart all of the VMs once things got into a hang condition.

Solution

In poking around, I found that the Service Integration Bus had almost a million messages and some of those were several YEARS old. I could see this from the WebSphere Admin Console, navigating to Service Integration->Buses->itim_bus->Messaging engines->ITIM_Application.000-itim_bus->Queue points. This showed me the queue depth of all queues. I then clicked on the one with the greatest depth (almost 500,000 messages), then clicked on the Runtime tab->Messages-> (first message). That showed me the timestamp of when the message was placed on the queue.

So I waited for a time when the system was quiet, then stopped WebSphere entirely on both nodes, then cleared the SIB following these instructions: https://www.ibm.com/docs/en/sig-and-i/10.0.2?topic=system-clearing-service-integration-bus. Then I started WebSphere on the primary node (the dmgr node), then the second node. With the SIB cleared out, the performance of both Application app servers was normal, with no excessive heap memory usage.

COMPLETELY cryptographically synchronizing two IBM Directory Servers

Background

This documentation exists to cryptographically sync two IBM Directory Server instances. That mostly works for the main LDAP directory data - essentially anything under the visible suffixes. HOWEVER, if you need to also cryptographically synchronize the data in the invisible "cn=Configuration" suffix, that's what this post is for.

The story of how I came across this issue is a normal one for a consultant. I was hired to help a customer work on an decade-old ISIM implementation that had been customized and configured by several different implementers. The customer needed to sunset their old AIX LDAP server and move the functionality to a new Red Hat Linux server. The existing server had multiple users defined with names like "cn=acmereadonly" or "cn=xyzservice", where there are multiple (but unknown and undocumented) integrations using these credentials, but no one actually knew the passwords. These users were all visible in ibmslapd.conf, with their AESS256-encrypted passwords, so it was extremely important to copy these users and their encrypted passwords to the new server so that existing integrations would continue to work.

There are actually TWO .KSF files

There are actually TWO .ksf files for an LDAP instance:

ibmslapddir.ksf - This one is used in the steps in the above link. It contains the encryption key and salt for encryption of data in the LDAP DIT. 

ibmslapdcfg.ksf - This one isn't mentioned at all in the the link above, and it is used for encryption of data in ibmslapd.conf, which includes the "hidden" cn=Configuration suffix.

Both of the above files are needed to accomplish my goal. 

What are the steps?

I'll start with the steps from the original link, clean them up a little and add the additional ones needed. 

To cryptographically synchronize two server instances, assuming that you created the first server instance do the following steps.

1. Create the second server instance, but do not start the server instance. do not run the idsbulkload command, or run the idsldif2db command on the second server instance.
2. Copy the ibmslapddir.ksf file (the ldap key stash file) AND the ibmslapdcfg.ksf file from the first server instance to the second server instance. The file is in the idsslapd-instance_name\etc directory on Windows systems, or in the idsslapd-instance_name/etc directory on AIX®, Linux, and Solaris systems. The instance_name is the name of the server instance.
3. Replace the ibmslapddir.ksf file AND the ibmslapdcfg.ksf file of the target server instance with the ibmslapddir.ksf file AND the ibmslapdcfg.ksf file of the source server instance. 
4. Edit the ibmslapd.conf file on the second server to set the value of ibm-slapdCryptoSync: under dn: cn=Configuration to be the exact same as the value on the first server.
5. If you have any users with passwords defined in ibmslapd.conf on the first server, you can copy those entries directly into the same place in the ibmslapd.conf file on the second server. They will be correctly loaded when the server starts.
6. Run the idsbulkload command from the second server instance.
7. Start the second server instance.
8. If you copied any users in step 5 above, try to log in with them on the second server instance, and you should be successful.

ldapsearch error: "scope is required for a null based search"

This command:

ldapsearch -D binduser@acme.com -w ? -h globalADcatalog -b DC=acme,DC=com cn=myuser

Will sometimes give the error:

scope is required for a null based search

But if you simply change directories (to /opt, for example), the command will run just fine.

This is due to filename expansion in the shell, where you happen to have a file (or directory) with a single-character name in your current working directory. The correct way to get around this is to change the command to:

ldapsearch -D binduser@acme.com -w \? -h globalADcatalog -b DC=acme,DC=com cn=myuser

FYI: The "-w ?" tells ldapsearch to prompt you for a password, and the backslash quotes the question mark so the shell doesn't try to interpret it. I like using this form of the command rather than typing the password on the command line for security purposes:

1. While the command is running, ANY user on the system can see the entire command line with the 'ps -ef' command.
2. If you're using a shared account (like a service account), any other people sharing that account can see all previous commands with the 'history' command.

Customizing IDI Linux Adapter - javax.naming.NameNotFoundException: isDollarAllowedInPwd

I was just looking at customizing the IDI Linux Adapter. I unzipped PosixLinuxProfile.jar and imported posixAdd.xml into my project in the IDI CE. When I clicked on the posixConn connector in the AssemblyLine, I got the error:

javax.naming.NameNotFoundException: isDollarAllowedInPwd 

I unzipped the PosixConnector.jar file and found the idi.inf file (that's not documented anywhere - this function is now performed with a tdi.xml file, but this older adapter still has this old file). In there, everything mainly looks good EXCEPT I noticed in the parameter section that isDollarAllowedInPwd didn't have a space before the following open curly brace, while every other parameter did. So I changed it from:

isDollarAllowedInPwd{

to

isDollarAllowedInPwd {

Then saved the idi.inf file, re-created the jar file, copied PosixConnector.jar to jars/connectors, and restarted IDI. Once I did that, I no longer got the error. I'm certain this doesn't affect the operation of the adapter, since this is simply for the Connector form in the CE, but I just didn't like seeing the error.

Additionally, another error you might see is a complaint about "_Default" and " Default". These are two properties files that are defined in all of the Linux Adapter assemblylines. The way to eliminate these errors (again, which only affect the CE; they don't affect the operation of the adapter) is to create two new tdiproperties files, one named "_Default" and one named " Default" (that is <space>Default). The GUI will let you create that weird name - just put a space in front of "Default". Then the CE won't complain about these two properties files any more.

There you go. Just wanted to share in case someone is encountering the problem.

Using IDI to convert ISIM audit XML data into HTML

 Related to my last post, I made a video covering the IDI assemblyline I wrote to convert ISIM audit XML data to HTML. Here's the video:

And here's the link to the Github repo with the project containing the assemblyline: https://github.com/franktate/IBM-Directory-Integrator

XSLT for ISIM Audit XML Data

 I made a video on using XSLT to convert ISIM's XML reconciliation audit data into more readable XML. Specifically, it takes XML that looks like this:

<?xml version="1.0" standalone="no"?>

<?xml-stylesheet type="text/xsl" href="frankaudit.xslt"?>

<Object name="Person">

  <Attribute name="title" type="String">

    <Scalar>Senior BA</Scalar>

  </Attribute>

  <Attribute name="acmegroups" type="String">

    <Scalar>group1</Scalar>

  </Attribute>

  <Attribute name="AttributeChangeOperation" type="Object">

    <Object name="AttributeChangeOperation.operation.replace">

      <Attribute name="acmegroups" type="String">

        <Scalar>group2</Scalar>

        <Scalar>group3</Scalar>

        <Scalar>group4</Scalar> 

      </Attribute>

    </Object> 

  </Attribute>

  <Attribute name="AttributeChangeOperation" type="Object">

    <Object name="AttributeChangeOperation.operation.remove">

      <Attribute name="title" type="String"></Attribute>

    </Object>

  </Attribute>

  <Attribute name="AttributeChangeOperation" type="Object">

    <Object name="AttributeChangeOperation.operation.add">

      <Attribute name="acmepositionnumber" type="String">

        <Scalar>99999999</Scalar>

      </Attribute>

    </Object>

  </Attribute>

</Object>

And turn it into HTML that renders like this:

The video is here: https://youtu.be/7yEkGZGN1Lg and the files used are available in my github repo here: https://github.com/franktate/ISIM-Audit-XML-XSLT