Using vi to edit files on an AWS instance can cause strange characters to be inserted

 This is a short one because I came across it the other day and I almost freaked out. I ssh'd into an AWS Alpine Linux instance using MobaXterm on Windows, and I edited a file with vi. I then saved it and re-opened it, and any lines that I added or modified were just messed up. I didn't get to the root of the issue, but I found that using the nano editor did NOT have this same problem. Strange, but wanted to document it in the hope that it may help someone else.

IBM Identity Manger recons hanging and apparent memory leak in JVM

Background

Two-node WebSphere cell, with two clusters - ITIM_Application and ITIM_Messaging, so two application servers per node (timapp1 and timmsg1, timapp2 and timmsg2). It's a fairly large system (6ooo services and 200000 person objects), and it has been running for almost a decade. It started as ISIM 6.0 and is now at version 10.0. The problem we initially saw was that scheduled recons would just go into a Pending state, and others wouldn't be started at all. The log files on both servers looked very similar, and nothing really stuck out, other than errors about hung threads. After more digging, I saw that timapp2, on the secondary server, would start and almost immediately start growing in heap memory size (I could see the memory usage in 'top' and also with 'jconsole'). Also, that application server would have very high CPU usage (200-500% on  a 6-cpu VM). The workaournd was to restart all of the VMs once things got into a hang condition.

Solution

In poking around, I found that the Service Integration Bus had almost a million messages and some of those were several YEARS old. I could see this from the WebSphere Admin Console, navigating to Service Integration->Buses->itim_bus->Messaging engines->ITIM_Application.000-itim_bus->Queue points. This showed me the queue depth of all queues. I then clicked on the one with the greatest depth (almost 500,000 messages), then clicked on the Runtime tab->Messages-> (first message). That showed me the timestamp of when the message was placed on the queue.

So I waited for a time when the system was quiet, then stopped WebSphere entirely on both nodes, then cleared the SIB following these instructions: https://www.ibm.com/docs/en/sig-and-i/10.0.2?topic=system-clearing-service-integration-bus. Then I started WebSphere on the primary node (the dmgr node), then the second node. With the SIB cleared out, the performance of both Application app servers was normal, with no excessive heap memory usage.

COMPLETELY cryptographically synchronizing two IBM Directory Servers

Background

This documentation exists to cryptographically sync two IBM Directory Server instances. That mostly works for the main LDAP directory data - essentially anything under the visible suffixes. HOWEVER, if you need to also cryptographically synchronize the data in the invisible "cn=Configuration" suffix, that's what this post is for.

The story of how I came across this issue is a normal one for a consultant. I was hired to help a customer work on an decade-old ISIM implementation that had been customized and configured by several different implementers. The customer needed to sunset their old AIX LDAP server and move the functionality to a new Red Hat Linux server. The existing server had multiple users defined with names like "cn=acmereadonly" or "cn=xyzservice", where there are multiple (but unknown and undocumented) integrations using these credentials, but no one actually knew the passwords. These users were all visible in ibmslapd.conf, with their AESS256-encrypted passwords, so it was extremely important to copy these users and their encrypted passwords to the new server so that existing integrations would continue to work.

There are actually TWO .KSF files

There are actually TWO .ksf files for an LDAP instance:

ibmslapddir.ksf - This one is used in the steps in the above link. It contains the encryption key and salt for encryption of data in the LDAP DIT. 

ibmslapdcfg.ksf - This one isn't mentioned at all in the the link above, and it is used for encryption of data in ibmslapd.conf, which includes the "hidden" cn=Configuration suffix.

Both of the above files are needed to accomplish my goal. 

What are the steps?

I'll start with the steps from the original link, clean them up a little and add the additional ones needed. 

To cryptographically synchronize two server instances, assuming that you created the first server instance do the following steps.

1. Create the second server instance, but do not start the server instance. do not run the idsbulkload command, or run the idsldif2db command on the second server instance.
2. Copy the ibmslapddir.ksf file (the ldap key stash file) AND the ibmslapdcfg.ksf file from the first server instance to the second server instance. The file is in the idsslapd-instance_name\etc directory on Windows systems, or in the idsslapd-instance_name/etc directory on AIX®, Linux, and Solaris systems. The instance_name is the name of the server instance.
3. Replace the ibmslapddir.ksf file AND the ibmslapdcfg.ksf file of the target server instance with the ibmslapddir.ksf file AND the ibmslapdcfg.ksf file of the source server instance. 
4. Edit the ibmslapd.conf file on the second server to set the value of ibm-slapdCryptoSync: under dn: cn=Configuration to be the exact same as the value on the first server.
5. If you have any users with passwords defined in ibmslapd.conf on the first server, you can copy those entries directly into the same place in the ibmslapd.conf file on the second server. They will be correctly loaded when the server starts.
6. Run the idsbulkload command from the second server instance.
7. Start the second server instance.
8. If you copied any users in step 5 above, try to log in with them on the second server instance, and you should be successful.

ldapsearch error: "scope is required for a null based search"

This command:

ldapsearch -D binduser@acme.com -w ? -h globalADcatalog -b DC=acme,DC=com cn=myuser

Will sometimes give the error:

scope is required for a null based search

But if you simply change directories (to /opt, for example), the command will run just fine.

This is due to filename expansion in the shell, where you happen to have a file (or directory) with a single-character name in your current working directory. The correct way to get around this is to change the command to:

ldapsearch -D binduser@acme.com -w \? -h globalADcatalog -b DC=acme,DC=com cn=myuser

FYI: The "-w ?" tells ldapsearch to prompt you for a password, and the backslash quotes the question mark so the shell doesn't try to interpret it. I like using this form of the command rather than typing the password on the command line for security purposes:

1. While the command is running, ANY user on the system can see the entire command line with the 'ps -ef' command.
2. If you're using a shared account (like a service account), any other people sharing that account can see all previous commands with the 'history' command.

Customizing IDI Linux Adapter - javax.naming.NameNotFoundException: isDollarAllowedInPwd

I was just looking at customizing the IDI Linux Adapter. I unzipped PosixLinuxProfile.jar and imported posixAdd.xml into my project in the IDI CE. When I clicked on the posixConn connector in the AssemblyLine, I got the error:

javax.naming.NameNotFoundException: isDollarAllowedInPwd 

I unzipped the PosixConnector.jar file and found the idi.inf file (that's not documented anywhere - this function is now performed with a tdi.xml file, but this older adapter still has this old file). In there, everything mainly looks good EXCEPT I noticed in the parameter section that isDollarAllowedInPwd didn't have a space before the following open curly brace, while every other parameter did. So I changed it from:

isDollarAllowedInPwd{

to

isDollarAllowedInPwd {

Then saved the idi.inf file, re-created the jar file, copied PosixConnector.jar to jars/connectors, and restarted IDI. Once I did that, I no longer got the error. I'm certain this doesn't affect the operation of the adapter, since this is simply for the Connector form in the CE, but I just didn't like seeing the error.

Additionally, another error you might see is a complaint about "_Default" and " Default". These are two properties files that are defined in all of the Linux Adapter assemblylines. The way to eliminate these errors (again, which only affect the CE; they don't affect the operation of the adapter) is to create two new tdiproperties files, one named "_Default" and one named " Default" (that is <space>Default). The GUI will let you create that weird name - just put a space in front of "Default". Then the CE won't complain about these two properties files any more.

There you go. Just wanted to share in case someone is encountering the problem.

Using IDI to convert ISIM audit XML data into HTML

 Related to my last post, I made a video covering the IDI assemblyline I wrote to convert ISIM audit XML data to HTML. Here's the video:

And here's the link to the Github repo with the project containing the assemblyline: https://github.com/franktate/IBM-Directory-Integrator

XSLT for ISIM Audit XML Data

 I made a video on using XSLT to convert ISIM's XML reconciliation audit data into more readable XML. Specifically, it takes XML that looks like this:

<?xml version="1.0" standalone="no"?>

<?xml-stylesheet type="text/xsl" href="frankaudit.xslt"?>

<Object name="Person">

  <Attribute name="title" type="String">

    <Scalar>Senior BA</Scalar>

  </Attribute>

  <Attribute name="acmegroups" type="String">

    <Scalar>group1</Scalar>

  </Attribute>

  <Attribute name="AttributeChangeOperation" type="Object">

    <Object name="AttributeChangeOperation.operation.replace">

      <Attribute name="acmegroups" type="String">

        <Scalar>group2</Scalar>

        <Scalar>group3</Scalar>

        <Scalar>group4</Scalar> 

      </Attribute>

    </Object> 

  </Attribute>

  <Attribute name="AttributeChangeOperation" type="Object">

    <Object name="AttributeChangeOperation.operation.remove">

      <Attribute name="title" type="String"></Attribute>

    </Object>

  </Attribute>

  <Attribute name="AttributeChangeOperation" type="Object">

    <Object name="AttributeChangeOperation.operation.add">

      <Attribute name="acmepositionnumber" type="String">

        <Scalar>99999999</Scalar>

      </Attribute>

    </Object>

  </Attribute>

</Object>

And turn it into HTML that renders like this:

The video is here: https://youtu.be/7yEkGZGN1Lg and the files used are available in my github repo here: https://github.com/franktate/ISIM-Audit-XML-XSLT

Decrypting the idisrv.sth file from IBM Directory Integrator

Background

The idisrv.sth file that's included in IBM Directory Integrator isn't a normal stash file. A normal stash file can be decrypted with a simple Perl script (see my previous post). However, this one is different. Happily, IBM does include a method to read this file, though it's hidden in a Jar file and requires some Java know-how.

Java Code

Here's the Java code required to read the file (in a file named ftStashFile.java - the name of the file must match the name of the class):

import com.ibm.di.server.StashFile;

import java.util.Vector;

public class ftStashFile extends StashFile {

  public static void main(String[] args) throws Exception {

    try {

      Vector thePassword = readPasswords();

      System.out.println(thePassword.firstElement());

    } catch (Exception e) {

      System.out.println("Exception encountered: " + e.toString());

    }

  }

}

The command to compile it is:

javac -cp .:/opt/IBM/TDI/V7.1.1/jars/common/miserver.jar ftStashFile.java

Then the command to run it is:

java -cp .:/opt/IBM/TDI/V7.1.1/jars/common/miserver.jar:/opt/IBM/TDI/V7.1.1/jars/common/tdiresource.jar:/opt/IBM/TDI/V7.1.1/jars/3rdparty/IBM/icu4j-4_4_2.jar ftStashFile

NOTE: You need to have the idisrv.sth file in the same directory as ftStashFile.class when you run it.

NOTE2: You will have problems if the JDK you're using is older than the JDK that was used to compile the IDI jar files. You'll get an error similar to this if that's the case:

  bad class file: ../jars/common/miserver.jar(com/ibm/di/server/StashFile.class)

    class file has wrong version 55.0, should be 52.0

    Please remove or make sure it appears in the correct subdirectory of the classpath.

And that's it. I've got a video going through how I figured this out here for more details.

JD-GUI Download: 

https://java-decompiler.github.io/

Perl script for decrypting a GSKit-created stash file:

https://blog.gulfsoft.com/2025/01/decrypting-properties-in-ibm-directory.html

Keystore Explorer (open-source tool similar to iKeyMan):

https://keystore-explorer.org/downloads.html

Decrypting properties in IBM Directory Integrator

If you look at the usage from the IDI encryption utility (cryptoutils.sh), you'll see this:

[root@jazzsm9 serverapi]# ./cryptoutils.sh 
CTGDKD446I Usage:
-input <input file>
-output <output file>
-mode <encrypt|decrypt|encrypt_config|decrypt_config>
-keystore <keystore file>
-storepass <keystore password>
-alias <encryption key alias>
[ -keypass <key password> ]
[ -transformation <encryption transformation> ]
[ -storetype <keystore type> ]
[ -cryptoproviderclass <security provider used for encryption> ]

Unfortunately, none of those "-mode" options will let you decrypt values in any of the *.properties files (e.g. global.properties, solution.propterties, etc.) So how do you do it?

To get the answer, you need to find the online documentation here to find that there are two additional options that aren't listed above. They are:

encrypt_props
decrypt_props

Once you know that, you're over the largest obstacle. But now you have several additional flags with values to provide, and the documentation doesn't give you an example of doing exactly this. So here's the example:

/opt/IBM/TDI/cev10/serverapi/cryptoutils.sh \

-input /opt/IBM/TDI/ftsoldir/solution.properties \

-output /tmp/foo \

-mode decrypt_props \

-keystore /opt/IBM/TDI/ftsoldir/testserver.jks \

-storepass server \

-alias server

In the above case, I wanted to decrypt the encrypted values in my solution.properties file. My solution directory is /opt/IBM/TDI/ftsoldir. Notice also that you MUST provide the certificate alias that points to the server certificate in the solution directory. By DEFAULT (meaning: all of this can be changed), the alias of that certificate is "server", it is stored in the $SOLDIR/testserver.jks keystore, and the password of the keystore is "server". The name of the keystore and the alias are specified in these two properties in solution.properties:

com.ibm.di.server.encryption.keystore = testserver.jks
com.ibm.di.server.encryption.key.alias = server

If, however, you forget the password, that's not a good thing. Normally you can decrypt an IBM stash file with a perl script like this:

#!/usr/bin/perl
use strict;
die "Usage: $0 <stash file>n" if $#ARGV != 0;
my $file=$ARGV[0];
open(F,$file) || die "Can't open $file: $!";
my $stash;
read F,$stash,1024;
my @unstash=map { $_^0xf5 } unpack("C*",$stash);
foreach my $c (@unstash) {
last if $c eq 0;
printf "%c",$c;
}
printf " ";

However, that doesn't work on the IDI stash file (idisrv.sth) because this isn't a GSKit stash file. From the docs:

The stash file contains the Server keystore password values encrypted with AES128 with a fixed key.

Check back later to find out later how to read this stash file - I think I've figured out how to decrypt it.

I did figure it out, and here's the video.

And NOTE: This only matters if you want to CHANGE the password of the keystore and keep the keystore. If all you need to do is add a server certificate to the keystore for a new web connection, you can simply use the "Get Certificate" button in the HTTP Server Connector to get the server's certificate and add it to the keystore.

IBM Identity Manger: Adding a new attribute

Background

The documentation on this is lacking on this topic, and the example LDIF file provided is in a format that will cause idsldapmodify to fail with a painfully obtuse error message. So I'm documenting this here to help the next person avoid some headache.

Details

Here's the documentation on how to add an attribute. In there, it provides this LDIF file as an example:

dn: cn=schema changetype: modify add: attributetypes attributetypes: ( myAttribute-oid NAME ( 'myAttribute' ) DESC 'An attribute I defined for my LDAP application' EQUALITY 2.5.13.2 SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 {200} USAGE userApplications ) - add: ibmattributetypes ibmattributetypes: ( myAttribute-oidDBNAME ( 'myAttrTable' 'myAttrColumn' ) ACCESS-CLASS normal LENGTH 200 )

However, if you try to import that LDIF file with the idsldapmodify command given, you'll get the following error:

ldapmodify: invalid format (line 5 of entry: cn=schema)

After a bit of pain, I figured out that the whitespace in the lines are the problem. You need to either concatenate the continued lines together OR have some white space at the front of the continued lines. Additionally, there should be a space in betwee "oid" and "DBNAME". So if you change the file to the following, you will no longer get the error:

dn: cn=schema
changetype: modify
add: attributetypes
attributetypes: ( myAttribute-oid NAME ( 'myAttribute' ) DESC 'An attribute I defined for my LDAP application'    EQUALITY 2.5.13.2 SYNTAX 1.3.6.1.4.1.1466.115.121.1.15    {200} USAGE userApplications )
-
add: ibmattributetypes
ibmattributetypes: ( myAttribute-oid DBNAME ( 'myAttrTable' 'myAttrColumn' ) ACCESS-CLASS normal LENGTH 200 )
What about updating the objectclass?
That's what I was wondering! Simply adding an attribute doesn't really help you much until you add it as a MUST or a MAY attribute to an objectclass. IBM has an almost complete guide to doing that here, But it is incomplete. For one thing, the format of the data is wrong again. The long attribute values should all be on a single line, like this:
dn: cn=schema
changetype: modify
replace: objectclasses
objectclasses: ( 2.16.840.1.113730.3.2.2 NAME 'inetOrgPerson' DESC 'Defines entries representing people in an organizations enterprise network.' SUP 'organizationalPerson' Structural MAY ( audio $ businessCategory $ carLicense $ departmentNumber $ employeeNumber $ employeeType $ givenName $ homePhone $ homePostalAddress $ initials $ jpegPhoto $ labeledURI $ mail $ manager $ mobile $ pager $ photo $ preferredLanguage $ roomNumber $ secretary $ uid $ userCertificate $ userSMIMECertificate $ x500UniqueIdentifier $ mytest ) )
Additionally, where did that value for "objectclasses" come from?? Well, that came from this command:
ldapsearch -b cn=schema -s base objectclass=* objectclasses | grep \'inetOrgPerson\'
Running that, you will get a result similar to:
objectclasses=( 2.16.840.1.113730.3.2.2 NAME 'inetOrgPerson' DESC 'Defines entries representing people in an organizations enterprise network.' SUP organizationalPerson STRUCTURAL MAY ( audio $ businessCategory $ carLicense $ departmentNumber $ displayName $ employeeNumber $ employeeType $ givenName $ homePhone $ homePostalAddress $ initials $ jpegPhoto $ labeledURI $ mail $ manager $ mobile $ o $ pager $ photo $ preferredLanguage $ roomNumber $ secretary $ uid $ userCertificate $ userPKCS12 $ userSMIMECertificate $ x500UniqueIdentifier ) )
Modify that to include your new attribute, then put that into the LDIF file. Then run idsldapmodify to import that LDIF file into your server. After that, you'll have your new attribute added to the inetOrgPerson objectclass.