Why don't databases have foreign keys any more?

If you've looked at a database from a vendor lately (Tivoli included; some example products that include databases are CCMDB, TADDM, ITCAM for WAS, ITCAM for RTT to name a few), you'll notice that there are very few (or NO) relationships between the tables. What this means is that you can't get a nice Entity Relationship Diagram from them, and that makes it quite a bit harder to write reports using the data in them. 

So why is it this way?

More and more, software developers are using Object-relational mapping (ORM) components to abstract the software from the database. Here is a pretty comprehensive list of object-relational mapping software. What this software does is abstract the data relationships from BOTH the software AND from the database, putting that relationship information into various files that are used by the selected ORM implementation. The impact this has is that it is difficult as a consumer/user of the software to write reports directly against the database - because the first step is having to reverse engineer the usage of the tables in the database.

What can be done about it?

I can think of a couple of approaches:

1. The best solution I can think of is that you need to ask the vendor for ERDs for all databases that will be used to store collected metrics. 

2. Since there may be some valid intellectual property-related arguments against no. 1 above, the next best approach would be to ask the vendor for the SQL needed to produce specific reports.

3. If neither of the above works, then reverse engineering is the only approach left. I've had success in this area by turning up the debugging on the software and looking for SQL "SELECT" statements in the log files. 

Data visualization using Google Spreadsheets, Yahoo Pipes and Google Maps

OK, so this isn't a Tivoli-related post, but since a big part of what we all want to do is visualize data, I thought that this post that I found through reddit.com was an exellent description of how to use several free web-based utilities to gather and display data. Here's the link:

http://ouseful.wordpress.com/2008/10/14/data-scraping-wikipedia-with-google-spreadsheets/

Here is the author's summary:

So to recap, we have scraped some data from a wikipedia page into a Google spreadsheet using the =importHTML formula, published a handful of rows from the table as CSV, consumed the CSV in a Yahoo pipe and created a geocoded KML feed from it, and then displayed it in a Yahoo map.

I haven't come up with an implementation that gathers Tivoli data, but I can think of one:

The main thing you would need is to get HTML that can be consumed by Google Spreadsheets. One way to do this would be with a JSP (on WAS) or PHP (on IBM HTTP Server) or other server-side language that makes SOAP or direct database calls to retrieve data from TDW and display it in HTML. Once you have that, you could then follow the steps in the article to viualize the data.

Creating TPM Signatures for Multi_sz registry entries

When creating some software signatures for Windows the use of registry entries to identify an application/operating system on a system. If you check out many of the existing Windows signatures, you will see that they use registry entries extensively.

One issue I came across was around the use of multi_sz type registry entries. This came up when scanning a Windows server, the operating system was not defined. In order to define the operating system the following keys were needed:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProductName=Microsoft Windows Server 2003
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\CSDVersion=Service Pack 1
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ProductOptions\ProductType=ServerNT

And the multi_sz
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ProductOptions\ProductSuite=Blade Terminal Server

For this last key, the Blade and Terminal Server were on separate lines. After looking at other signatures that also used multi_sz, I could not see what was required to separate the lines, so I tried a space, \n and a couple others that never worked.

I then tried the dcmexport command and looked at other multi_sz definitions. They were useless as the character was not a normal ASCII character.

Then I found Windows signatures located in $TIO_HOME/eclipse/plugins/windows-operating-system/xml. In there I found the code & # 1 4 5 ; (remove the spaces) used in existing definitions. I then created the software signature, associated it with the Software Definition Windows Server 2003 Standard Edition SP1 (for lack of a better one for now), ran a scan and the operating system was recognized.

Hope this helps.

TCR Report Packages

Tivoli Common Reporting provides two ways to package your BIRT report design files viz. Report Design Format and Report Definition Format. Both have the same acronym, so we can't abbreviate them.  This article explains the two formats in detail.

Report Design Format

This is the simplest of the two formats. How do you create them?   Simply create a zip file of your BIRT project directory (note! you need to create the zip file of the PROJECT not the ReportDesign file).  That's it. In Windows, you can right click on the project directory and click Send To->Compressed (Zipped) Folder, there you're done!  Import the resulting zip file into TCR and it will import the files with .rptdesign extension as TCR reports. 

The Path to the report design files will be reflected in the TCR navigation tree.  For example, if your report design files are located in /GBSReports/Netcool directory, then the report designs in TCR will appear under GBSReports->Netcool in the TCR Navigation tree. 

Report Definition Format

TCR Documentation does not give good idea about Report Definition Format. Creating Report Definition Format involves creating an XML file that describes the report set structure. The advantage of this format is that you can create a well documented report set and it also provides the ability to share the same report design across multiple levels in the navigation tree.   Even though, this format helps to create a well-documented report set, ironically, TCR does not have a detailed documentation such as details about the XML Schema.   The OPAL ITM62 Report set uses the Report Definition Format and it should give you some idea, but that's all I have come across so far.

Overall, the Report Design Format should be good enough for most of our needs and it is much easier to change the Report set structure as well.  However, Report Definition Format gives more control over Report set organization and one hopes that more information about the format will be available in future releases.

And here's how to use extended attributes in TADDM

UPDATE: Since TADDM 7.1.2 is MUCH friendlier about extended attributes, I changed the title of this post. Details on reporting on extended attributes to come soon.

... you'll want to be able to populate them during discovery. And this requires the use of the SCRIPT: directive in your custom server template. The documentation on this is pretty sketchy, so that's why I'm including a full script here that will set the value of the "FrankCustom" custom attribute for a LinuxComputerSystem.

Prerequisites
1. You must create the extended attribute named FrankCustom for the LinuxUnitaryComputerSystem class (or any class in its inheritance chain).

2. You must enable the LinuxComputerSystemTemplate custom Computer System sensor. You can do this from the TADDM GUI, Discovery drawer, click on Computer Systems, and edit the LinuxComputerSystemTemplate to set it to "enabled".

3. You have to create a file named $COLLATION_HOME/etc/templates/commands/LinuxComputerSystemTemplate with the following line:

SCRIPT:/myscripts/myscript.py

If it's a Jython script you're calling (which it is in this case), the script filename MUST end with a ".py" extension. If you try to use ".jy", it will absolutely fail.

Now, you need to create the file /myscripts/myscript.py with the following contents:



import sys
import java
from java.lang import System

coll_home = System.getProperty("com.collation.home")
System.setProperty("jython.home",coll_home + "/external/jython-2.1")
System.setProperty("python.home",coll_home + "/external/jython-2.1")

jython_home = System.getProperty("jython.home")
sys.path.append(jython_home + "/Lib")
#sys.path.append(coll_home + "/lib/sensor-extensions")
sys.prefix = jython_home + "/Lib"

import traceback
import string
import re
import StringIO
import jarray

from java.lang import Class
from com.collation.platform.util import ModelFactory
from java.util import Properties
from java.util import HashMap
from java.io import FileInputStream
from java.io import ByteArrayOutputStream
from java.io import ObjectOutputStream


#Get the osobject and the result object
os_handle = targets.get("osobject")
result = targets.get("result")
system = targets.get("system")
# two more pieces of information available
#env = targets.get("environment")
#seed = targets.get("seed")

# Execute a command (it's best to give the full path to the command) and
# store its output in the variable named output
output = os_handle.executeCommand("/bin/ls -l /tmp")
s_output = StringIO.StringIO(output)
output_list = s_output.readlines()

# This loop just stores the last line of output from the above command in
# the variable named "output_line"
for line in output_list:
output_line = line
s_output.close()

#Get the AppServer ModelObject for the MsSql server
#app_server = result.getServer()

#Set up the HashMap for setting the extended attributes
jmap=HashMap()
jmap.put("FrankCustom",output_line)
bos=ByteArrayOutputStream()
oos=ObjectOutputStream(bos)
oos.writeObject(jmap)
oos.flush()
data=bos.toByteArray()
oos.close()
bos.close()

#Call setExtendedAttributes on the AppServer
system.setExtendedAttributes(data)


And there you have it. The examples you find in the documentation and on the TADDM wiki just won't lead you to any kind of success because they're missing some very key information.

Feel free to use extended attributes in TADDM

Update: TADDM 7.1.2 has massive improvements in data accessibility, so this post is no longer valid.


Don't let the documentation lead you astray - extended attributes in TADDM should be your LAST CHOICE for customization!

I realize that's a bold statement (especially since there's so much information about creating extended attributes in the documentation), but I'll explain why it's a valid statement:

1. You can't run any reports based on values stored in extended attributes! None. You can say, for example "give me all ComputerSystems where myCustomAttribute contains the string 'foo'". Just can't do it. The only call to get extended attribute values requires the GUID (or GUIDs) of the object(s) you're interested in. This is enough reason to stay away from extended attributes.

2. Extended attributes don't get imported into the CCMDB unless you do a LOT of customization to the TADDM integration adapter.

If you have custom data that you need to store in TADDM, what you need to do is become familiar with the CDM (Common Data Model) and the APIs available in TADDM. They are documented in three different archives that you have to extract (these all exist on your TADDM server):

CDMWebsite.zip - contains information about the CDM, including entity relationship diagrams.
model-javadoc.tar.gz - contains javadoc information for interacting with model objects.
oalapi-javadoc.tar.gz - contains javadoc information for accessing TADDM, from which you may just output XML, or you might obtain references to model objects, whose methods are defined in the above file.

If you're reading these files on your own workstation, make sure to grab a fresh copy from the TADDM server after each fixpack, as Tivoli does update this documentation.

By reading and understanding the documentation above, you can determine the appropriate object types to use to effectively store all of your data.

Getting the TADDM Discovery Client (aka TADDM GUI) to run when you have Java 1.5 and 1.6 installed

Getting the TADDM Discovery Client (aka TADDM GUI) to run when you have Java 1.5 and 1.6 installed

The TADDM GUI uses the Java Web Start technology (i.e. a JNLP file) to run. This allows the application to be launched as a full-screen application (rather than just an applet) over the web. If you ONLY have Java 5 installed , there shouldn’t be a problem. But if you have 1.5 AND 1.6 installed, you’ll probably get an error stating something similar to:

[SunJDK14ConditionalEventPump] Exception occurred during event dispatching:

java.lang.ClassCastException: java.lang.StringBuffer cannot be cast to java.lang.String

at com.collation.gui.client.GuiMain.setApplicationFont(GuiMain.java:511)

at com.collation.gui.client.GuiMain.doLogin(GuiMain.java:442)

at com.collation.gui.client.GuiMain.access$100(GuiMain.java:83)

The reason the problem occurs is that the .JNLP file states that it wants Java 1.5+, which means 1.5 or higher, so javaws will pick the highest one available, which is 1.6. But 1.6 isn't supported by the TADDM GUI. So here's how to fix it.

This file describes what has to be done to fix the problem. I apologize for the PDF link, but trying to get all of the screenshots uploaded was driving me crazy.