Interesting Visualization

Anyone in IT cares about data at some point, so I try to bookmark visualizations that I think are noteworthy. The one I saw today is:

http://www.michaelvandaniker.com/labs/browserVisualization/

I'm not interested in the data that's behind it; I just think it's a neat way to present it.

http://manyeyes.alphaworks.ibm.com/manyeyes/

This is IBM's community Many Eyes project, which can be used by anyone.

http://code.google.com/apis/visualization/documentation/gallery.html

This is Google's Visualization API, also open to anyone.

http://graphviz.org/

Graph/node/relationship visualization.

GbsNcoSql User Guide

You can download the complete documentation for GbsNcoSql from the following link.

 

http://www.gulfsoft.com/downloads/GbsNcoSqlUserGuide.pdf

 

Hope you find it useful.

Now you have an alternative to FreeTDS - GbsNcoSql

Thanks to Venkat Saranathan, who developed this tool and wrote the description. He also wrote 'gbscmd' for use with ITM 6.x.

Many times, you need to analyze the Omnibus data using scripts. For this purpose, Netcool Omnibus provides nco_sql utility that lets you run the Omnibus queries from the command line. However, one of the main limitations of the nco_sql utility is that it is nearly impossible to parse the output. The difficulties include single record spanning across multiple lines, no delimiter option to name a few.

We at Gulf Breeze worked on this requirement sometime back and developed an utility to address some of these shortcomings. The utility is written in Java and can be run with the Java Runtime that comes with Omnibus. Some of the benefits/features of this utility includes the following.

• Currently, it can run any Omnibus "SELECT" queries only. Though the utility could potentially be modified to run other queries, the idea is to use the nco_sql utility for all other type of queries.
• Consistent one-record per line output
• Ability to specify your own delimiter with "-d" switch.
• Ability to suppress header output with "-n" switch.
• Ability to specify queries in a file or read from command line.
• No need to compile any FreeTDS drivers. It comes with the necessary FreeTDS drivers.
• Authorization information kept in a separate file and can be specified with -a switch. You don't need to specify the password in your scripts.
• Platform independent and works with the IBM and Sun JREs.

Here is a sample usage of this utility.

usage: java -jar GbsNcoSql.jar -a [-n] [-d ] [-f | -q ]

Where,

-a File name containing authorization information
-d Delimiter
-f File name containing SQL query text
-h Displays the help information
-n No header output
-q Query text

Here is a sample data.

$ java -jar GbsNcoSql.jar -a my.auth -q "SELECT Node,Tally from alerts.status"
Node,Tally
sys1,1
sys21,1
sys3,1
192.168.1.50,190

You can create an authorization manually in text editior. Here is how my authorization file looks. Please note that in the value for server is the hostname NOT the object server name.

$ cat my.auth

server=somehost

port=4100
user=root
password=mypass

Click this link to download.

Perl one-liner to convert ITM timestamps to normal timestamps

One of the annoying aspects of ITM troubleshooting is that the timestamps in the logs are written in epoch time in hexadecimal instead of 'human-readable' format. I don't know the exact reason behind it, but the whole purpose of logs is to aid in troubleshooting and timestamps are critical piece of information in troubleshooting.

Anyway, to convert the log timestamps to normal timestamp, I have been using the following one-liner.

perl -lane 'if (/^(.)([\dA-F]+)(\..*)/) { printf "%s%s%s\n", $1, scalar(localtime(oct("0x$2"))),$3; }' <log-file>

The one-liner can be used to read from pipe as well as below.

tail -100 <log-file> perl -lane 'if (/^(.)([\dA-F]+)(\..*)/) { printf "%s%s%s\n", $1, scalar(localtime(oct("0x$2"))),$3; }'

Hope this makes troubleshooting little easier.

GBSCMD V3.6.4 - Overview

GBSCMD is a free Gulf Breeze offering for performing ITM operations from command line. It is complimentary to tacmd tool and performs many operations that are not provided by tacmd. GBSCMD uses ITM Webservices feature extensively and some of the benefits of GBSCMD are listed below.

• Feature support for backlevel ITM versions. Most of the features works from ITM 6.1 on wards.
• Provides a way to execute remote commands on agents.
• Provides a way to get agent data in CSV format.
• Clear offline managed systems from Command line.
• Provides a postemsg like feature to send events to situation event console.

If you would like to learn about GBSCMD tool, here are some links to the past articles about GBSCMD. You can google search "gbscmd site:blog.gulfsoft.com".

http://blog.gulfsoft.com/2008/03/gbscmd-v35-new-features.html

http://blog.gulfsoft.com/2008/03/simple-postemsg-like-solution-for-itm_15.html

We have been tweaking the tool from time to time to introduce new features and address bug fixes and this blog discuss some of the changes introduced during the recent versions. The latest and greatest version of the tool as of this writing is version 3.6.4. If you would like to get the latest version, please feel free to contact me at venkat at gulfsoft.com or Tony Delgross at gulfsoft.com. If you have some suggestion on features that you would like to see in GBSCMD, please feel free to write to me as well.

Here are the changes introduced to GBSCMD since version 3.6.1

Version 3.6.1

This version introduced support for row filtering of SOAP Call results with --afilter option. The following example uses --afilter option to get the disk data from an agent and filters the results to include only C: drive information.

./gbscmd ct_get --auth itm62.auth --Object NT_Logical_Disk --target Primary:ITM62:NT --afilter Disk_Name;EQ;C:

You can also get the last 24 hour history data from the agent using --history switch to CT_Get. Some of the other changes include ability to see the results in XML format using --xml option (Thanks IV for making that change!).

Version 3.6.2

This version introduced new SOAP calls for starting and stopping situations at RTEMS level using --starttemssit and --stoptemssit options. If you are running backlevel of ITM and would like the ability to start and stop situations, you can use this feature. It also addressed some timeout issues while getting large SOAP data. (e.g. listing of all managed systems in the entire enterprise for a large ITM Setup).

Version 3.6.3

This version uses new SOAP call for listing situations running on individual agents.

Version 3.6.4

This version introduces column filtering of SOAP results with --attribute option. For example, if you want to get the disk data and only interested in Disk Name and Free Megabytes attributes, the following command line will get you the information.

./gbscmd ct_get --auth itm62.auth --Object NT_Logical_Disk --target Primary:ITM62:NT --attribute Disk_Name --attribute Free_Megabytes

This version also adds --version switch to identify the current GBSCMD version.

The complete documentation is available for download from the following link.

http://www.gulfsoft.com/downloads/gbscmdReferenceManual.pdf

If you have any questions or suggestions, please feel free to mention them in the comments section.

How to get ITM agent data in CSV format using GBSCMD

Here is a commonly asked question from our customers. How do I get the real time agent data exported to excel format (CSV) from command line for further analysis? Even though this can be done from Portal using a combination of logical workspaces and table view, it is real simple  with gbscmd tool.

 

With GBSCMD, you just invoke ct_get subcommand providing agent name and attribute group you're interested in.  For example, if you want to get the list of current disk usage information from a Windows OS agent, you can use the following command.

 

./gbscmd ct_get --auth itm62.auth --Object NT_Logical_Disk --target Primary:ITM62:NT  >> disk_usage.csv

 

If you would like to get this information on set of windows agents, you can easily loop thru them one after another like the example below.

 

for agent in `cat myagents.list`
do
 ./gbscmd ct_get --auth itm62.auth --Object NT_Logical_Disk --target $agent  >> disk_usage.csv
done
   
Hope you find this tip useful.

Multiple Logfile Monitoring Agent - KG2

As promised, here is a link to download a logfile monitoring agent. It will work on Unix, Linux or Windows and can monitor up to 10 logfiles per instance of the agent. This is a very generic agent with no filtering. The agent will take up to the first 1024 characters and put them into one field. As always, I've run this in the lab it works great - your setup may vary. The watchdog is setup and the memory threshold is set at 100mb for all OS platforms.

Step 1 - Download and unzip. The entrie package is 26mb in size, it contains all of the supported platforms.

ZIP version

http://www.gulfsoft.com/downloads/blog_downloads/KG2.zip

or the tar/gzip version

http://www.gulfsoft.com/downloads/blog_downloads/KG2.tgz


Step 2 - Run the installIraAgentTEMS.bat or .sh on your HUB TEMS and Remote TEMS.
Step 3 - Run the installIraAgentTEPS.bat or .sh on your TEP server (recycle the TEPS afterwards)
Step 4 - Populate your depot using tacmd addbundles -i /path... or just copy the zip file to the destination and run the installIraAgent.bat or .sh.
Step 5 - Configure it. Create an instance and add at least file name to the agent.
Step 6 - Once the agents appear in your portal server, you can create a situation that uses the "Scan for String within String" method to search for specific keywords.

Step 7 - Have fun.