Some possible responses to TADDM Error CTJTD3602E

IBM's documentation contains the following description of this error:

CTJTD3602E: The Change Manager is still processing. Wait and retry the discovery at a later time.

Explanation: The Change Manager is still running to process recent changes discovered. The discovery cannot be started until the Change Manager completes.

Operator Response: None.

Administrator Response: Allow time for the Change Manager to complete its processing before starting a discovery.

Programmer Response: None.

What we have found is that this condition can possibly occur for at least a couple of different reasons. The two situations we've found have been:

1. There was a deadlock on the database that we had to clear. A "little while" after we cleared the lock, we were able to successfully run a discovery again without receiving the error message.

2. The change manager partially ran, but didn't update the CHANGE_SERVER_STATUS table. Specifically, it left the value of the STATUS column set to 16 for the last discovery. To fix this, we had to run the following SQL:

update DB2TADDM.CHANGE_SERVER_STATUS set status = 17

This updates the STATUS column for all rows in the table. You could limit it with a WHERE clause, but in our case, it was valid as is.

We then needed to stop and restart TADDM, and the error message went away.

Identifying newly added systems in Tivoli Monitoring using GBSCMD

Hope you have been using the GBSCMD utility developed by Gulf Breeze for Tivoli Monitoring product.

 

In the latest version of GBSCMD (V3.7.1), there is a cool new feature added to the program. With "gbscmd listnewsystems", it can list the recently deployed systems in the environment along with the deployed time and the type of the agent.  With this feature, you can setup "after-install" policy scripts in ITM  just like Framework with little scripting.

 

Note: IV Blankenship wrote a blog article (http://www.gulfsoft.com/blog_new/print.php?sid=322) about this idea sometime back. 

 

Please note that the list of new systems is lost when TEMS is recycled, so you need to make sure this information is saved in a persistent store like a database or flatfile.

 

You can download the latest documentation for GBSCMD from the following link.

 

http://www.gulfsoft.com/downloads/gbscmdReferenceManual.pdf

 

Not yet subscribed to GBSCMD. Please contact us from your company email and we will send you a copy. Don't worry we will use the email only to send you future updates to GBSCMD. For those, who already subscribed to GBSCMD, we will send an update this week.

 

Are you going to Pulse 2010?
Come see Gulf Breeze at Pulse so we can show why we are the best.

Interesting Visualization

Anyone in IT cares about data at some point, so I try to bookmark visualizations that I think are noteworthy. The one I saw today is:

http://www.michaelvandaniker.com/labs/browserVisualization/

I'm not interested in the data that's behind it; I just think it's a neat way to present it.

http://manyeyes.alphaworks.ibm.com/manyeyes/

This is IBM's community Many Eyes project, which can be used by anyone.

http://code.google.com/apis/visualization/documentation/gallery.html

This is Google's Visualization API, also open to anyone.

http://graphviz.org/

Graph/node/relationship visualization.

GbsNcoSql User Guide

You can download the complete documentation for GbsNcoSql from the following link.

 

http://www.gulfsoft.com/downloads/GbsNcoSqlUserGuide.pdf

 

Hope you find it useful.

Now you have an alternative to FreeTDS - GbsNcoSql

Thanks to Venkat Saranathan, who developed this tool and wrote the description. He also wrote 'gbscmd' for use with ITM 6.x.

Many times, you need to analyze the Omnibus data using scripts. For this purpose, Netcool Omnibus provides nco_sql utility that lets you run the Omnibus queries from the command line. However, one of the main limitations of the nco_sql utility is that it is nearly impossible to parse the output. The difficulties include single record spanning across multiple lines, no delimiter option to name a few.

We at Gulf Breeze worked on this requirement sometime back and developed an utility to address some of these shortcomings. The utility is written in Java and can be run with the Java Runtime that comes with Omnibus. Some of the benefits/features of this utility includes the following.

• Currently, it can run any Omnibus "SELECT" queries only. Though the utility could potentially be modified to run other queries, the idea is to use the nco_sql utility for all other type of queries.
• Consistent one-record per line output
• Ability to specify your own delimiter with "-d" switch.
• Ability to suppress header output with "-n" switch.
• Ability to specify queries in a file or read from command line.
• No need to compile any FreeTDS drivers. It comes with the necessary FreeTDS drivers.
• Authorization information kept in a separate file and can be specified with -a switch. You don't need to specify the password in your scripts.
• Platform independent and works with the IBM and Sun JREs.

Here is a sample usage of this utility.

usage: java -jar GbsNcoSql.jar -a [-n] [-d ] [-f | -q ]

Where,

-a File name containing authorization information
-d Delimiter
-f File name containing SQL query text
-h Displays the help information
-n No header output
-q Query text

Here is a sample data.

$ java -jar GbsNcoSql.jar -a my.auth -q "SELECT Node,Tally from alerts.status"
Node,Tally
sys1,1
sys21,1
sys3,1
192.168.1.50,190

You can create an authorization manually in text editior. Here is how my authorization file looks. Please note that in the value for server is the hostname NOT the object server name.

$ cat my.auth

server=somehost

port=4100
user=root
password=mypass

Click this link to download.

Perl one-liner to convert ITM timestamps to normal timestamps

One of the annoying aspects of ITM troubleshooting is that the timestamps in the logs are written in epoch time in hexadecimal instead of 'human-readable' format. I don't know the exact reason behind it, but the whole purpose of logs is to aid in troubleshooting and timestamps are critical piece of information in troubleshooting.

Anyway, to convert the log timestamps to normal timestamp, I have been using the following one-liner.

perl -lane 'if (/^(.)([\dA-F]+)(\..*)/) { printf "%s%s%s\n", $1, scalar(localtime(oct("0x$2"))),$3; }' <log-file>

The one-liner can be used to read from pipe as well as below.

tail -100 <log-file> perl -lane 'if (/^(.)([\dA-F]+)(\..*)/) { printf "%s%s%s\n", $1, scalar(localtime(oct("0x$2"))),$3; }'

Hope this makes troubleshooting little easier.

GBSCMD V3.6.4 - Overview

GBSCMD is a free Gulf Breeze offering for performing ITM operations from command line. It is complimentary to tacmd tool and performs many operations that are not provided by tacmd. GBSCMD uses ITM Webservices feature extensively and some of the benefits of GBSCMD are listed below.

• Feature support for backlevel ITM versions. Most of the features works from ITM 6.1 on wards.
• Provides a way to execute remote commands on agents.
• Provides a way to get agent data in CSV format.
• Clear offline managed systems from Command line.
• Provides a postemsg like feature to send events to situation event console.

If you would like to learn about GBSCMD tool, here are some links to the past articles about GBSCMD. You can google search "gbscmd site:blog.gulfsoft.com".

http://blog.gulfsoft.com/2008/03/gbscmd-v35-new-features.html

http://blog.gulfsoft.com/2008/03/simple-postemsg-like-solution-for-itm_15.html

We have been tweaking the tool from time to time to introduce new features and address bug fixes and this blog discuss some of the changes introduced during the recent versions. The latest and greatest version of the tool as of this writing is version 3.6.4. If you would like to get the latest version, please feel free to contact me at venkat at gulfsoft.com or Tony Delgross at gulfsoft.com. If you have some suggestion on features that you would like to see in GBSCMD, please feel free to write to me as well.

Here are the changes introduced to GBSCMD since version 3.6.1

Version 3.6.1

This version introduced support for row filtering of SOAP Call results with --afilter option. The following example uses --afilter option to get the disk data from an agent and filters the results to include only C: drive information.

./gbscmd ct_get --auth itm62.auth --Object NT_Logical_Disk --target Primary:ITM62:NT --afilter Disk_Name;EQ;C:

You can also get the last 24 hour history data from the agent using --history switch to CT_Get. Some of the other changes include ability to see the results in XML format using --xml option (Thanks IV for making that change!).

Version 3.6.2

This version introduced new SOAP calls for starting and stopping situations at RTEMS level using --starttemssit and --stoptemssit options. If you are running backlevel of ITM and would like the ability to start and stop situations, you can use this feature. It also addressed some timeout issues while getting large SOAP data. (e.g. listing of all managed systems in the entire enterprise for a large ITM Setup).

Version 3.6.3

This version uses new SOAP call for listing situations running on individual agents.

Version 3.6.4

This version introduces column filtering of SOAP results with --attribute option. For example, if you want to get the disk data and only interested in Disk Name and Free Megabytes attributes, the following command line will get you the information.

./gbscmd ct_get --auth itm62.auth --Object NT_Logical_Disk --target Primary:ITM62:NT --attribute Disk_Name --attribute Free_Megabytes

This version also adds --version switch to identify the current GBSCMD version.

The complete documentation is available for download from the following link.

http://www.gulfsoft.com/downloads/gbscmdReferenceManual.pdf

If you have any questions or suggestions, please feel free to mention them in the comments section.