Troubleshooting ITM Event Synchronization

ITM Event Synchronization (aka SUF, Situation Update Forwarder, SitForwarder) with Omnibus is mainly straightforward, but there are a couple of things to look out for.

Setting the logLevel

The 3 available logging levels in C:\IBM\SitForwarder\etc\situpdate.conf are:

low
med
verbose

Where the Logfile is Created

The default log file is in the Omnibus log directory, HOWEVER, the tracing log file is in:

C:\tmp\itmsynch\logs

This is where you'll see more information if you increase the logging level - NOT in the default log.

What to Look for in the Log

If event updates are correctly being sent back to ITM (for example, when you acknowledge an ITM event from the AEL in Omnibus), the following is an example of what you should see in the log file (with logLevel=verbose):

2009.06.03 16:24:09.703-04:00 com.tivoli.candlenet.SituationUpdateForwarder formatEvent IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP Entry, parm 1 = op=a,sn=NT_Invalid_Logon_Attempt,sh=webappprod,sv=itmserver.gulfsoft.com:,et=,di=SYSTEM

2009.06.03 16:24:09.703-04:00 com.tivoli.candlenet.SituationUpdateForwarder formatEvent IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP Number of tokens is 6

2009.06.03 16:24:09.703-04:00 com.tivoli.candlenet.SituationUpdateForwarder formatEvent IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP Display item token is SYSTEM

2009.06.03 16:24:09.703-04:00 com.tivoli.candlenet.SituationUpdateForwarder formatEvent IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP SOAP string: sysadminxxxxxNT_Invalid_Logon_Attemptwebappprod

2009.06.03 16:24:09.703-04:00 com.tivoli.candlenet.SituationUpdateForwarder formatEvent IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP Exit, return value = sysadminxxxxxNT_Invalid_Logon_Attemptwebappprod

2009.06.03 16:24:09.703-04:00 com.tivoli.candlenet.SituationUpdateForwarder getServerUrlConnection IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP Entry, parm 1 = op=a,sn=NT_Invalid_Logon_Attempt,sh=webappprod,sv=itmserver.gulfsoft.com:,et=,di=SYSTEM

2009.06.03 16:24:09.703-04:00 com.tivoli.candlenet.SituationUpdateForwarder getServerUrlConnection IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP Default port 3661 is being used by Web Services.

2009.06.03 16:24:09.703-04:00 com.tivoli.candlenet.SituationUpdateForwarder sendEvent IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP Entry


Additionally, you should have no lines data in the C:\IBM\SitForwarder\persistence\situpdate_* file.

What it Looks like When Things are Wrong

In my case, I was acknowledging an event that was sent from ITM to Omnibus BEFORE I got the SUF configured, so the correct EIF Probe rules were not in place. These lines were in the persistence file:

op=a,sn=NT_Invalid_Logon_Attempt,sh=webappprod,sv=itmserver.gulfsoft.com:,et=,di=SYSTEM

Notice that there is a ":" (colon) on the end of the itmserver hostname. That is WRONG. The line SHOULD look like:

op=a,sn=NT_Invalid_Logon_Attempt,sh=Primary:SVRCITRIXXT01:NT,sv=itmserver.gulfsoft.com:3661,et=,di=SYSTEM

Notice the ":3661" at the end of the host name.

Here's an example of the lines in the log file when I had this problem:

2009.06.03 16:22:49.376-04:00 com.tivoli.candlenet.SituationUpdateForwarder formatEvent IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP SOAP string: sysadminxxxxxNT_Invalid_Logon_Attemptwebappprod

2009.06.03 16:22:49.376-04:00 com.tivoli.candlenet.SituationUpdateForwarder pollCacheFile IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP Exception while processing cache file: null

2009.06.03 16:22:49.376-04:00 com.tivoli.candlenet.SituationUpdateForwarder pollCacheFile IBM Tivoli Monitoring TEC Synchronization netcoolserver.gulfsoft.com IP java.lang.NullPointerException
at com.tivoli.candlenet.SituationUpdateForwarder.sendEvent(Unknown Source)
at com.tivoli.candlenet.SituationUpdateForwarder.pollCacheFile(Unknown Source)
at com.tivoli.candlenet.SituationUpdateForwarder.poll(Unknown Source)
at com.tivoli.candlenet.SituationUpdateForwarder.main(Unknown Source)
at com.tivoli.candlenet.W32Service$SUFRunnable.run(Unknown Source)
at java.lang.Thread.run(Thread.java:568)


Notice that there are no lines stating that the event is being sent.