Thursday, January 2, 2025

IDI Operations: Always Start the IDI Server and CE with the "-s" Flag

Keeping it simple, you should always start the ibmdisrv or ibmditk processes with the "-s" flag to specify the solution directory to use. If you don't do this, then you're just using whatever happens to be set in your environment, which could lead to quite a bit of confusion. As an example, you could have your CE (the eclipse IDI GUI) running with a DIFFERENT solution directory than your Default ibmdisrv process. This could lead to a case where properties that resolve perfectly in the GUI don't resolve when you go to run an AL. You can avoid this by always using full paths to properties files, but then that makes your solutions much less portable.

IDI Operations: Finding the Solution Directory of a Running ibmdisrv Process

Background

I am working with a client that has had ITIM and TDI/IDI installed and running for over a decade, with different groups responsible for the administration of each. They now have around ten ibmdisrv processes running, with some of them configured to run the same Listener ALs (so the one that starts first wins). None of the servers are started with the "-s" flag, so it's a little difficult to figure out their solution directories.

Each ibmdisrv and ibmditk process needs to have a Solution Directory (aka SOLDIR, SolDir, or current working directory) defined. All default logging is done in the solution directory, and all relative paths used are relative to the solution directory. Basically, the solution directory is very important, and this post will show you how you can find it for a running process.

The solution shown here is for Linux, because that is the most common OS I've seen for running IBM Directory Integrator. The second most popular platform is Windows. If you're running Windows, you can get similar data using the Process Explorer GUI tool.

Solution

The first thing to look at is the process listing for the IDI server java process. The easiest way to find all of these processes running on your server is with this command:

# ps -ef | grep IDILoader

root 5693 5683 0 Jan01 pts/0 00:07:57 /opt/IBM/TDI/ceV10/jvm/jre/bin/java -cp /opt/IBM/TDI/ceV10/IDILoader.jar -Dlog4j2.configurationFile=file:etc/log4j2.xml --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED com.ibm.di.loader.ServerLauncher -d

We see from that output that the PID of this java process is 5693 and the Parent PID (PPID) is 5683. The most straightforward way I've found to find the SolDir for this server is using this command:

# lsof -Pp 5693 | grep cwd

The output should be similar to:

java 5693 root cwd DIR 253,0 4096 68731158 /opt/IBM/TDI/prod_V7.1.1

Easy as that, we see that the SolDir for this server is /opt/IBM/TDI/prod_V7.1.1.

Additional Details

We can get more details about the PPID with this command:

# ps -fwwp 5683

UID PID PPID C STIME TTY TIME CMD

root 5683 5462 0 Jan01 pts/0 00:00:00 /bin/sh /opt/IBM/TDI/ceV10/ibmdisrv -d

What we see here is that the /opt/IBM/TDI/ceV10/ibmdisrv script was called with only the "-d" flag to start this process. When that script runs, it calls the bin/setupCmdLine.sh script, which eventually calls the script bin/defaultSolDir.sh to set the solution directory of the process. (A great post about this and more can be found at this link.) While this is truly great information to know, one drawback is that the solution directory can be overridden with the TDI_SOLDIR environment variable. That's why I like the lsof approach better on Linux. The behavior of lsof on AIX is different and doesn't point to the solution directory like it does on Linux, so this approach may be better there. In either case, it's good to know both.

Tuesday, December 24, 2024

Logging in IBM Directory Integrator

Introduction

I've been creating a bunch of AssemblyLines in IDI/TDI/ISVDI recently, and wanted to share a trick to help you set up logging to a file whose path includes BOTH the name of the project and the name of the AssemblyLine.

Quick and Dirty

1. In the Log Settings for the AL, make sure to add a log. I prefer FileRollerAppender, but you choose whatever you want.

2. Click on File Path and set it to Advanced (JavaScript)

3. Set the value to the following:

var logName = "TOBEREPLACEDATRUNTIME";
try {
    logName = task.getParent().getName() + "/" + task.getShortName();
} catch(e) {}
return "/opt/IBM/TDI/logs/" + logName + ".log";

4. Click OK and you're ready to go. The logfile for the AL will be:

/opt/IBM/TDI/logs/project_name/AL_name.log

Longer and More Details

The Quick and Dirty section above is for advanced/experienced users who have been working with the product for a while and have been beating their heads against a wall trying to find this solution (maybe a handful of people out there). This section is for everyone else.

Background

IDI has tons and tons and tons of logging options. So many, in fact, that most customers will just stick with the default, which logs everything to ibmdi.log, which can make life difficult because all ALs are logged in the exact same file, with their messages intertwined. There can absolutely be benefits to that (i if you're using something like Splunk for centralized log storage, for example). To me, it's much easier to have a different log file for each AssemblyLine. And because you can easily end up with hundreds of ALs in your implementation, it makes sense to me to have one directory per Project, just to make it easier to work through the directory structure.

The specific situation I'm in is that my client has had ITIM and TDI running for over a decade, with different people managing the implementation over the years. This means that they have multiple TDI servers (at various versions) running on the same host. If I was to set this up from scratch starting right now, an easier way to accomplish something VERY similar is to add these two lines to solution.properties for the DI server:

SystemLog.defaultCreateLog=true
SystemLog.defaultMaxGenerations=10

This will cause each AL to log messages in files named:

$SOLDIR/system_logs/$project/$ALName/<logfile>.<timestamp>.log (10 max)

But that's if I was starting from scratch, which I'm not. So back to my original solution.

Walkthrough

Here's an animated GIF walking you through the process of adding a log to an AL, then configuring that log to have a name that includes the project and AL name.

Explanation

There are a couple of interesting parts that I think deserve a little more detail.

try/catch

Without the try/catch block in the code, you'll see an error when going to edit the logger, like this:

This is true for any form element in IDI that you implement as JavaScript. If you reference the task or work or conn objects, you'll get a big red error similar to the one above. That's why I first declare the variable logName with a string value. Then inside the try block, I set it to the value I want to see at runtime. The catch block does nothing, but syntactically it's required.

getParent()

So where did I find the getParent() method of the task object? It's in the API javadocs, which you can find on your own system in YOUR_TDI_INSTALLDIR/api or you can find them online here. The task object is described in the AssemblyLine class (com.ibm.di.server.AssemblyLine specifically; there is another one named AssemblyLine, but it is com.ibm.di.api.jmx.mbeans.AssemblyLine, which is not the same).

Why did you change the Pattern for the logger?

I removed "[%c]" from the pattern because this information takes up a TON of visual space and is 100% unnecessary when using a different log file for each AL.

What to do next?

So here's a bonus strategy that can save you a lot of time as you create new ALs. Set this for all of your existing ALs (or at least those for which it makes sense). Then, when you copy any AL, the log will already be dynamically set for you in your new AL. In my experience, this is extremely useful because I end up creating quite a few "utility" ALs for performing one-off operations. I always start by copying an existing AL, then giving it a new name and modifying it as appropriate for the task at hand. I generally keep these utility ALs in a single project named something like "CLIENT_NAME_Utilities".

Comments

In every one of my ALs, I enable the Prolog - Before Init hook specifically as the place where I provide some documentation for the AL for the next ~~sucker~~ developer that comes along and has to maintain it.

Parting thoughts

If you got something out of this, please share the link to get it more exposure.

Friday, August 30, 2024

Working with dates in IBM Security Directory Integrator

Working with dates in TDI can be tricky, since you'll normally be dealing with not only ISIM's internal format, but also various formats from different databases and other sources. The easiest way I've found is to use the java.time.* classes that were introduced in Java 8. Specifically, the LocalDateTime class is great because it has methods like minusHours(), minusDays(), etc. to make it easy to get a time from "X time ago".

Here are some examples I've come up with to use with the different formats I've run into on my latest contract:

var ldtObj = java.time.LocalDateTime.now();

var dtfObj = java.time.format.DateTimeFormatter.ofPattern("yyyyMMdd");

var today = ldtObj.format(dtfObj);

// today is "20240830" for August 30, 2024

var dtfObjHHmm = java.time.format.DateTimeFormatter.ofPattern("yyyyMMddHHmm");

var todayHHmm = ldtObj.format(dtfObjHHmm);

// todayHHmm is "202408301531" for 15:31 on August 30, 2024

var dtfObjH = java.time.format.DateTimeFormatter.ofPattern("yyyy-MM-dd:HH");

var todayH = ldtObj.format(dtfObjH);

// todayH is "2024-08-30:15" for 15:00 on August 30, 2024

var dtfObjChars = java.time.format.DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH'hrs'mm'mins'");

var todayChars = ldtObj.format(dtfObjChars);

// todayChars is "2024-08-30T15hrs31mins" for 15:31 on August 30, 2024

// just put single quotes in the Pattern around any literal characters you want in the resulting string.

var utcZone = java.time.ZoneId.of("UTC");

var ldtObjUtc = java.time.LocalDateTime.now(utcZone);

var dtfObjZ = java.time.format.DateTimeFormatter.ofPattern("yyyyMMddHHmm'Z'");

var todayZ = ldtObjUtc.format(dtfObjZ);

// todayZ is "202408301531Z" for 15:31 on August 30, 2024 Zulu (UTC) time.

// get the time "3 hours ago"

var ldtObjThreeHoursAgo = ldtObj.minusHours(3);

var dtfObj = java.time.format.DateTimeFormatter.ofPattern("yyyyMMddHHmm");

var today = ldtObj.format(dtfObj);

// today is "202408301231" for three hours before 15:31 August 30, 2024 (so 12:31)

Friday, August 9, 2024

Update erlaststatuschangedate in ISIM with a TDI HR Feed

IBM has a support article on this topic here:

https://www.ibm.com/support/pages/how-update-erlaststatuschangedate-through-tdi-hr-feed#:~:text=If%20the%20%22erpersonstatus%22%20attribute%20value%20is%20being%20modified,Work%20attribute%20called%20%22erlaststatuschange%22%20to%20the%20Input%20Map

Unfortunately, the code provided is incorrect and will throw an exception if you try to use it.

Here's the corrected code:

// set up variables to set the erlaststatuschangedate attribute

// get milliseconds since epoch in local time zone.

var mydate = new Date().valueOf();

// get milliseconds of offset from Zulu/UTC

var localOffset = new Date().getTimezoneOffset() * 60000;

// add/subtract milliseconds based on local timezone to get Zulu time.

var myTZ = mydate + localOffset;

// create a new Date() object representing "now" in Zulu time.

var utcDate = new Date(myTZ);

// Format the date into a string in the correct format

var myTZStr = new java.text.SimpleDateFormat("yyyyMMddHHmm").format(utcDate);

// Add "Z" to the end of the string to complete the correct format.

var myTZStrZ = myTZStr + "Z";

// set the value of erlaststatuschangedate

work.setAttribute("erlaststatuschangedate",myTZStrZ);

Monday, July 22, 2024

IBM Directory Integrator error "unknown format type at"

So here's a strange one that I didn't find anywhere in my numerous searches, so I figured I would post it here. When running a TDI AssemblyLine, I got the following exception / error:

12:14:43,274 ERROR - [My-JDBC-Connector] CTGDIS810E handleException - cannot handle exception , addonly

java.lang.IllegalArgumentException: unknown format type at

at com.ibm.icu.text.MessageFormat.makeFormat(MessageFormat.java:2086)

at com.ibm.icu.text.MessageFormat.applyPattern(MessageFormat.java:589)

at com.ibm.icu.text.MessageFormat.<init>(MessageFormat.java:437)

at com.ibm.icu.text.MessageFormat.format(MessageFormat.java:1137)

at com.ibm.di.util.ParameterSubstitution.substitute(ParameterSubstitution.java:229)

at com.ibm.di.util.ParameterSubstitution.substitute(ParameterSubstitution.java:173)

at com.ibm.di.util.ParameterSubstitution.substitute(ParameterSubstitution.java:245)

at com.ibm.di.connector.JDBCConnector.putEntry(JDBCConnector.java:1202)

at com.ibm.di.server.AssemblyLineComponent.executeOperation(AssemblyLineComponent.java:3351)

at com.ibm.di.server.AssemblyLineComponent.add1(AssemblyLineComponent.java:2012)

at com.ibm.di.server.AssemblyLineComponent.add(AssemblyLineComponent.java:1965)

at com.ibm.di.server.AssemblyLine.msExecuteNextConnector(AssemblyLine.java:3780)

at com.ibm.di.server.AssemblyLine.executeMainStep(AssemblyLine.java:3391)

at com.ibm.di.server.AssemblyLine.executeMainLoop(AssemblyLine.java:2989)

at com.ibm.di.server.AssemblyLine.executeMainLoop(AssemblyLine.java:2972)

at com.ibm.di.server.AssemblyLine.executeAL(AssemblyLine.java:2939)

at com.ibm.di.server.AssemblyLine.run(AssemblyLine.java:1317)

12:14:43,276 ERROR - CTGDIS266E Error in NextConnectorOperation. Exception occurred: java.lang.IllegalArgumentException: unknown format type at