The Cylance Smart Antivirus agent will ruin your day

I am currently helping a customer move their ITM 6 infrastructure from AIX to Red Hat 8, and the largest hurdle has been the Cylance agent. When doing any kind of enterprise install, my first step is to copy the install files to all of the servers (in this case it is 16 servers: 2 HUB TEMS, 12 RTEMS, 2 TEPS). In its default configuration, the Cylance agent will remove files that it determines are suspicious. In my case, that means that it deleted one or two tar files, and would re-delete them whenever I copied them over again. The cylance log under /opt/cylance/desktop/log showed exactly what it was doing, so we were able to work with the Cylance team to correct this.

After the delete issue was resolved, we found that the Cylance agent was stopping some executables from running, with just a "Segmentation fault" error, and the error still existed after stopping the Cylance agent. This is because even though the agent wasn't running, it has hooks into kernel system calls that leverage a local cache. That took a while to resolve, but we finally got all of the appropriate directories whitelisted.

The last problem encountered was with the Cylance agent's Memory Protection feature. In this case, it caused 'tacmd tepslogin' to fail with a bunch of text to the command line and no information in the normal ITM logs. Looking in the Cylance log file again, I could see that it was blocking some memory action performed by the ITM java executable. That now seems to be resolved.

Hopefully this short post can help others identify these types of issues before throwing their server out the window.

How to get live metrics for CPU, Memory,Disk etc by running db2 queries in the tdw server.

I just received this question today from an ITM user and thought it would be best to answer it here.

First, The "Getting Started with IBM Tivoli Monitoring on Distributed Environments" PDF contains great information on ITM 6.1, and it is applicable to all of the 6.x versions:

http://www.redbooks.ibm.com/redbooks/pdfs/sg247143.pdf

Chapter 4 specifically goes over the Tivoli Data Warehouse, including the DB2 table and column names. Specifically, look at section 4.2.4 for descriptions of the different table names.

The tricky part in DB2 is that the table and column names are all case sensitive, which means you need to enclose the table name inside double-quotes in your sql statement. For example, you would use this SQL statement to get percent processor time for all processes for all servers:

select "%_Processor_Time","Process_Name" from "NT_Process"

If you want to see data that hasn't yet made it to the TDW, then you need to use the krarloff command described here:

https://www.ibm.com/support/pages/itm-agent-insights-using-krarloff-utility-view-binary-short-term-historical-data-0

APM historical data without a TEMS

IBM APM only directly supports the storage of up to 32 days of data. You've always had the option to store older data in the ITM 6 data warehouse, but setting up an entire ITM 6 infrastructure has always seemed like a whole lot of work. As it turns out, you don't have to set up the entire infrastructure! You oonly have to set up a warehouse proxy agent, summarization and pruning agent and the warehouse database. No TEMS or TEPS, as the WPA and SPA can work in autonomous mode.

Madhavan Vyk recently posted a great article on DeveloperWorks detailing exactly how to configure this:

https://www.ibm.com/developerworks/community/blogs/0587adbc-8477-431f-8c68-9226adea11ed/entry/Integrating_IBM_Cloud_APM_8_x_with_Tivoli_Data_Warehouse?lang=en

What to use instead of ITMSuper

ITMSuper is a JavaScript based tool that can be used for maintaining the health of your ITM 6.x environment. It was written by IBM and made available as a separate download, but was never completely supported. It's even less supported today, as it only supports Internet Explorer 8. Here is a great blog post from Shaun R at IBM pointing to different tools that you should use instead, all written by IBM's own John Alvord:

https://www.ibm.com/developerworks/community/blogs/0587adbc-8477-431f-8c68-9226adea11ed/entry/Helping_us_help_you_ITM_Bitesize_Edition_ITMSuper?lang=en

Live Video Presentation: ITM Nugget: Overview of how to retrieve enterprise wide information in one simple output

IBM's Mark Leftwich has made a great video on how you can pull enterprise wide information in one simple step. These techniques and examples are great for audits, sanity checking monitoring and problem diagnosis work:

Video - Automated Application Management using IBM Tivoli System Automation and IBM Tivoli Monitoring

Video - a tutorial on the use of the ITM TEPS

Video - IBM Tivoli Monitoring (ITM) for POWER Systems AIX and VIO

Getfile, Putfile, Executecommand - added in ITM 6.2.2 fp02

Enable this functionality (Check out Venkat’s earlier post)

GETFILE:

{-m|--system} SYSTEM
{-s|--source} REMOTE_FILE
{-d|--destination} LOCAL_FILE
[{-t|--type} Transfer MODE] - Either [text|bin]
[{-f|--force}] – Forces an overwrite of the file if it exists

Example:
tacmd getfile -m itmtest:LZ -s /tmp/itmtest.log -d /base_logs/itmtest_08272010.out -t text –f
--------------------------------------------------------------


PUTFILE:

{-m|--system} SYSTEM
{-s|--source} LOCAL_FILE
{-d|--destination} REMOTE_FILE
[{-t|--type} MODE] ] - Either [text|bin]
[{-f|--force}] – Forces an overwrite of the file if it exists

Example:

tacmd putfile -m itmtest:LZ -s /automationscripts/ testing.sh -d /tmp/testing.sh -t text –f

NOTE: When transferring the file, it sets the permissions of the file to “666”.

--------------------------------------------------------------


EXECUTECOMMAND:
tacmd executecommand

{-m|--system} SYSTEM
{-c|--commandstring} COMMAND_STRING
[{-w|--workingdir} REMOTE_WORKING_DIRECTORY}]
[{-o|--stdout}]
[{-e|--stderr}]
[{-r|--returncode}]
[{-l|--layout}]
[{-t|--timeout} TIMEOUT]
[{-d|--destination} LOCAL_STD_OUTPUT_ERROR_FILENAME]
[{-s|--remotedestination} REMOTE_STD_OUTPUT_ERROR_FILENAME]
[{-f|--force} FORCE_MODE}
[{-v|--view}]

Example:

tacmd executecommand -m itmtest:LZ -c "/tmp/testing.sh" -o -e -r -l –v


NOTE: You will need to parse the STDOUT from this command to get the local STDERR, STDOUT, Return Code etc…

Using Custom SQL Queries with MSLs to Get Real-Time Data from Agents

Thanks to Venkat for this tip:

Here is an example SQL statement you can use in a query to pull real-time NT Logical Disk data from all of the agents that have been added to the "MYWIN" Managed System List:

SELECT ORIGINNODE,TIMESTAMP,INSTCNAME,PCUSED FROM KNT.WTLOGCLDSK AT ("HUB_ITMSERVER") WHERE SYSTEM.PARMA("NODELIST", "MYWIN",5 )

In this statement, "HUB_ITMSERVER" is the label (not the hostname) of the TEMS the agents are connected to, "MYWIN" is the MSL name, and the "5" is simply the length of the string "MYWIN".

To find other available table names, the easiest way is to look in the agent application support SQL files on the TEMS.

Creating a Web Services/SOAP client in eclipse

The Eclipse development environment is very powerful, so I figured I would create a simple SOAP client in it from a given WSDL file. As it turns out, it's a little painful, so I wanted to write up the steps I had to go through.

I used Eclipse 3.3.2 (Europa) JEE: Eclipse 3.3.2 (Europa) JEE on Windows XP in a VM

I started with an empty Java project named "FirstJava".

First import your WSDL file into your project.
- Select the "src" folder of your proejct, right-click and select "Import"
- Choose General/File System
- Select the folder containing your file and click OK
- now place a check mark next to your WSDL file and click OK

Now download Tomcat. There is a built-in "J2EE Preview" that works for some things, but not for this.

Now create a new Target Runtime and server associated with Tomcat.
- Select the project and select Project->Properties, then select "Targeted Runtimes" on the left of the dialog.



- Click the "New" button
- Select Apache->Tomcat 5.5 AND ALSO CHECK "Also create new local server"



- Input the appropriate values as needed.


- Select the newly-create runtime


- Click OK to save the properties.

Create a new Web Services Client
- Right-click your WSDL file name and select New->Other..., then in the dialog displayed, select Web Services->Web Service Client).



- Your WSDL file name should be filled in at the top of the dialog


- In this dialog, use the slider underneath "Client Type" to specify "Test client" (move it all the way to the top).
- Click Finish.
- This will create a bunch of new code in your current project, plus it will create a new project (named "FirstJavaSample" in my case) with the JSPs you'll be able to (hopefully) run to test your client.


- This will give you an error about the JSP not supporting org.apache.axis.message.MessageElement[]. Just click OK several times until the error box goes away. We'll fix that later.

If all went well, you should see something like the following:


Now we have to fix the errors.

Create a JAR file named FirstJava.jar containing the contents of the bin directory of your FirstJava project.

Copy that file to the Tomcat "Common/lib" folder (C:/Apache/Tomcat5.5/Common/lib on my system).

You will additionally need to find these files under the eclipse/plugins directory and copy them to the Tomcat Common/Lib folder:

axis.jar
saaj.jar
jaxrpc.jar
javax.wsdl15_1.5.1.v200705290614.jar
wsdl4j-1.5.1.jar

(If you can't find them on your system, use Google to find and download them. One of them - I don't recall which - was tricky to find for me because it was actually in another JAR file.)

Now stop the Tomcat server by selecting it in the "Servers" view and clicking the stop (red square) button.


Now re-run your application by opening the FirstJavaSample project and finding the file named "TestClient.jsp". Right-click that file and select Run As->Run On Server, select your Tomcat server and click Finish.

You should now see that things are working correctly.


You may need to edit the generated JSP files to add input fields and such, but that's specific to your particular file.

Good luck, and happy coding.

Customizing ITM 6.1 tep.jnlp to work with Java 1.5

The default tep.jnlp file works great in certain environments, and not at all in others. Basically, it works if you only have Java 1.4.2 installed, or if you have Java 1.4.2 and Java 1.5 installed. But things don't work so well if you have Java 1.6 installed. This change fixes that problem.

What I did was change the line that reads:

j2se version="1.4+" size="64m" args="-showversion -noverify"

to the following:

j2se version="1.5*" size="64m" args="-showversion -noverify"

If you save this file with a new name, you must also update the line in the file that references itself (i.e. change 'href="tep.jnlp"' to 'href="newfilename.jnlp"').

And I found that ONE of these two files (the original or the changed) will work for almost all users. They just need to try one, and if that doesn't work, try the other. The problem is that the Java 1.4.2 browser plugin doesn't support the changed syntax. So you can just tell your users to try the first, then the second.

Edit: If a user only has Java 1.6 installed, the changed file will work and will automatically download and install Java 1.5.