Wednesday, January 29, 2014

MongoDB Setup and Monitoring with Application Insight

Introduction

MongoDB might be new to you.  It certainly was new to us, so when we took some time to experiment with IBM SmartCloud Monitoring - Application Insight of a MongoDB, our first hurdle was setting up MongoDB!  It should be noted that stand-alone DB instances are not currently supported by the monitoring agent, so we needed to build up a clustered/replicated MongoDB configuration.  This is common in a production environment, but a bit more than we expected to need in the lab.

MongoDB Setup Overview

Our MongoDB setup was accomplished using a single virtual machine.  While this isn't a recommended setup for a production environment, it is perfectly fine for this testing.  Our setup included the following overall steps:

  1. Create 3 databases, each configured to use unique paths and ports (ports: 37017, 37018, 37019)
  2. Connect to the "soon-to-be" primary database (port 37017) and configure the replica set.
  3. Add second (port 37018) and third (port 37019) databases to the replica set.
  4. Installed a demo database using JSON import
  5. Started a Configuration Server (port 37020)
  6. Started mongos, pointing at Configuration Server and listening on port 37021
  7. Connected to mongos (port 37021)
  8. Added replica set to Shard
  9. Enabled Sharding on our demodb
  10. Enabled Sharding on the collection (aka table)

A special thanks to the following website for providing some demo database data and a nice presentation on MongoDB and Ruby apps.  Also thanks to the MongoDB docs for creating a cluster.

You may ask, why 3 databases, aren't 2 enough?  This is because the recommended MINIMUM number of databases for a Replica Set is 3.  If your setup only has 2 databases, Application Insight will flag the Global MongoDB Status as a Warning.

MongoDB Setup

VM hostname is "openpulse".
Replica Set is named "rs1".

Download MongoDB (we used 64-bit Linux) from here.
Untar file.  Location of files will be referred to as $MEDIA
The remaining commands assume the $MEDIA/bin directory are in your path.

mkdir -p /srv/mongodb/rs1-0 /srv/mongodb/rs1-1 /srv/mongodb/rs1-2
# Start up 3 databases
mongod --port 37017 --dbpath /srv/mongodb/rs1-0 --replSet rs1 --smallfiles --oplogSize 128
mongod --port 37018 --dbpath /srv/mongodb/rs1-1 --replSet rs1 --smallfiles --oplogSize 128
mongod --port 37019 --dbpath /srv/mongodb/rs1-2 --replSet rs1 --smallfiles --oplogSize 128

# Connect to to-be-primary
mongo --port 37017

#Give Primary a configuration for a replication set
rsconf = {
           _id: "rs1",
           members: [
                      {
                       _id: 0,
                       host: "openpulse:37017"
                      }
                    ]
         }

#Initiates the replica set
rs.initiate( rsconf )

#Displays Replication set
rs.conf()

#Add 2nd db - NOTE: you might need to wait for prompt to change to PRIMARY before continuing.
rs.add("openpulse:37018")

#Add 3rd db
rs.add("openpulse:37019")

#Display Replication set, confirm three are listed
rs.conf()

#Install DemoDB
curl -L http://c.spf13.com/OSCON/venuesImport.json | mongoimport --port 37017 -d demodb -c venues

#Start a config server
mkdir /data/configdb2
mongod --configsvr --dbpath /data/configdb2 --port 37020

#Start mongos pointing at config server
mongos --port 37021 --configdb openpulse:37020

#Connect to mongos
mongo --host openpulse --port 37021

#Add replica set to shard
sh.addShard( "rs1/openpulse:37017" )

#Shows replica set and other stats
db._adminCommand("connPoolStats");

#Connect to mongos
mongo --host openpulse --port 37021

#Enable sharding for the DB
sh.enableSharding("demodb")

#switch DB
use demodb

#Set index for sharding
db.venues.ensureIndex( { _id : "hashed" } )
db.venues.getIndexes()

#Setup sharding of the collection (table)
sh.shardCollection("demodb.venues", { "_id": "hashed" })


MongoDB Agent Setup

Reference IBM documentation here.
Our monitoring instance is named "Pulse21"

mongodb-agent.sh install

mongodb-agent.sh config pulse21
Agent configuration started...
Edit "Monitoring Agent for MongoDB" settings? [ 1=Yes, 2=No ] (default is: 1):

Agent Configuration :
Configuration variables for agent start up

The directory where the Java runtime is located. This value is optional. The agent will use the built in Java by default. Use this to over ride the Java to be used.
Java (default is: ):
Allows the user to say whether these system is a cluster or single replication set. This value is optional. By default the agent will monitor a cluster.
Type [ 1=Cluster, 2=Single Set ] (default is: 1):
Port Number for the router of a MongoDB cluster or a mongod instance of a MongoDB replication set being monitored. This value is optional. The agent will automatically discover the cluster location if only one is present on the system. This is used to over ride discovery of a cluster or explicitly monitor a replication set.
Port (default is: ): 37021  (Note: this is the port of the mongos process, not the individual databases)
The ip address for the host system of the router or the mongod instance. This value is optional. The agent will automatically discover the default ip address. This is used to select a particular interface on a system where MongoDB is bound to only one of several addresses.
Host (default is: ):

Agent configuration completed...
As a reminder, you should restart appropriate instance(s) for new configuration settings to take effect.

./mongodb-agent.sh start pulse21
Processing. Please wait...
Starting Monitoring Agent for MongoDB ...
Monitoring Agent for MongoDB started

Application Insight Screenshots




Thanks,
Anthony Segelhorst and Jamie Carl

Monday, January 6, 2014

Creating ITM deployment packages for IBM Endpoint Manager



A question came up on the IBM Endpoint Manager forum on how to deploy ITM agents using IEM. This is not something that I have done in IEM, but have done with TCM (Configuration Manager) and TPM (Provisioning Manager). I did recall that ITM had a switch that allowed for the creation of SPD (Software Package Definitions) or SPB (Software Package Blocks), so thought I would look at this for IEM.
Now there are a few ways of using the SPB/SPD or even just create the bundle with no export format. If you use the SPD format, you will get the command line that would need to be used in the IEM action script. So for this example. I am going to use the SPD export. Just to note though, IEM also supports the import of SPB files, so this could be another option.

Assumption: The particular bundle that you desire is already in the ITM depot.

The high level steps would be as follows:
    - Create the bundle
    - Create IEM Download File
    - Determine the sha1 information.
    - Review the SPD file and determine install command and options for the NT_Silent_Install.txt
    - Use the BFArchive tool to ZIP the bundle directory and upload to the IEM server.
    - Determine relevance method
    - Create the fixlet

Create the Bundle

ITM uses the “tacmd” command with the “exportBundles” switch to create a silent installation folder. For this example, I am using the “-o SPD” switch to define that I want the export to also create the SPD file to determine the command line an contents of the NT_Silent_Install.txt file.
tacmd exportBundles -o SPD -t nt -e e:\data\bundle
KUICEB023I: Are you sure you want to export the following bundles to the e:\data\bundle directory?
Product Code : NT
Deployable   : True
Version      : 063000000
Description  : Monitoring Agent for Windows OS
Host Type    : WINNT
Host Version : WINNT
Prerequisites:

Product Code : NT
Deployable   : True
Version      : 063000000
Description  : Monitoring Agent for Windows OS
Host Type    : WIX64
Host Version : WIX64
Prerequisites:
 KUICEB024I: Enter Y for yes or N for no: y

 KUICEB020I: Exporting bundles to the e:\data\bundle directory. The time required to complete this operation depends on the number and size of the exported bundles.

 KUICEB022I: The following bundles were successfully exported to the e:\data\bundle directory:

Product Code : NT
Deployable   : True
Version      : 063000000
Description  : Monitoring Agent for Windows OS
Host Type    : WINNT
Host Version : WINNT
Prerequisites:

Product Code : NT
Deployable   : True
Version      : 063000000
Description  : Monitoring Agent for Windows OS
Host Type    : WIX64
Host Version : WIX64
Prerequisites:

Create IEM Download File

BFArchive-8.0.0.0.exe -a e:\data\bundle\NT_063000000_WINNT e:\data\NT_063000000_WINNT.tmp

Determine sha1 information

The SHA1 tool (https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Endpoint%20Manager/page/SHA1%20Tool) can be used to determine the SHA1 value and also will provide a template for the “prefetch” command.

Sha1.exe -r e:\data\NT_063000000_WINNT.tmp
prefetch NT_063000000_WINNT.tmp sha1:32685b95f970e8f61e187d8433360072f213dba8 size:85360201 http://EXAMPLEURL/REPLACEME.exe

Use the sha1 value as the folder name and create a folder under “E:\Program Files (x86)\BigFix Enterprise\BES Server\wwwrootbes\Uploads\” (this is the directory on my TEM server), then  copy the TMP file to the new folder. For this example the folder would be “E:\Program Files (x86)\BigFix Enterprise\BES Server\wwwrootbes\Uploads\32685b95f970e8f61e187d8433360072f213dba8”

Review the SPD file and determine install command and options for the NT_Silent_Install.txt


If you do not already have a silent install response file, this can be created by reviewing the information in the SPD file. This is no different than the response file used for non-IEM deployments. This also goes for the command line.
The SPD file does contain quite a bit of information in it and if you are not familiar with them, you might be better off reading the ITM install guides on how to create the response file.
Most of what is needed is either in the “default_variables” or in the “add_text_file_objects”. These sections will show the variables that are used and the command line required to execute the install.

default_variables
     LOG_MODE = "LOGMODE"
     SOURCE_DIR = "e:\data\bundle\NT_063000000_WINNT"
     TEMS_HOSTNAME = "localhost"
     CANDLEHOME = "C:\IBM\ITM"
     FIREWALL = "NO"
     NETWORK_PROTOCOL = "IP.PIPE"
     IP_PIPE_PORT_NUMBER = "1918"
     CANDLE_ENCRIPTION_KEY = "IBMTivoliMonitoringEncryptionKey"
     ARGS_INSTALL_CMD = '/c start /wait $(CANDLEHOME)\spbInstall\setup.exe /w /z"/sf$(CANDLEHOME)\spbInstall\NT_Silent_Install.txt" /s  /f2"$(DEST_SILENT_INSTALL_LOG)"'
     DEST_SILENT_INSTALL_LOG = "$(temp_dir)\Silent_NT.log"
     DEST_SILENT_UNINSTALL_LOG = "$(temp_dir)\Silent_NT.log"
     ARGS_UNINSTALL_CMD = '/c start /wait $(CANDLEHOME)\spbInstall\setup.exe /w /z"/sf$(CANDLEHOME)\spbInstall\NT_Silent_Uninstall.txt" /s  /f2"$(DEST_SILENT_UNINSTALL_LOG)"'
end

add_text_file_objects
     replace_if_existing = y
     file = $(CANDLEHOME)\spbInstall\NT_Silent_Install.txt
     stop_on_failure = n

     add_command_line
        replace_if_existing = y
        text = ";Install Directory=C:\IBM\ITM"
        command = "Install Directory"
        key = "C:\IBM\ITM"
        position = begin
        pattern = "Install Directory=C:\IBM\ITM"
     end

     add_command_line
        replace_if_existing = y
        text = ";EncryptionKey=IBMTivoliMonitoringEncryptionKey"
        command = "EncryptionKey"
        key = "IBMTivoliMonitoringEncryptionKey"
        position = begin
        pattern = "EncryptionKey=IBMTivoliMonitoringEncryptionKey"
     end

     add_line
        replace_if_existing = y
        text = "Install Directory=$(CANDLEHOME)"
        position = before_first
        pattern = ";Install Directory=C:\IBM\ITM"
     end
     add_line
        replace_if_existing = y
        text = "EncryptionKey=$(CANDLE_ENCRIPTION_KEY)"
        position = before_first
        pattern = ";EncryptionKey=IBMTivoliMonitoringEncryptionKey"
     end
     add_line
        replace_if_existing = y
        text = "[CMA_CONFIG]"
        position = end
     end
     add_line
        replace_if_existing = y
        text = "FTO Flag=N"
        position = end
     end
     add_line
        replace_if_existing = y
        text = "AUDIT=Y"
        position = end
     end
     add_line
        replace_if_existing = y
        text = "Protocol1=$(NETWORK_PROTOCOL)"
        position = end
     end
     add_line
        replace_if_existing = y
        text = "IPPIPE_Port=$(IP_PIPE_PORT_NUMBER)"
        position = end
     end
     add_line
        replace_if_existing = y
        text = "IPPIPE_Host=$(TEMS_HOSTNAME)"
        position = end
     end
     add_line
        replace_if_existing = y
        text = "CONNECT=Y"
        position = end
     end
   end

With this information, I am able to build the action script. Of course there will be different paths for some items, but that goes with knowing the IEM environment.

My command line will look something like this:
waithidden "C:\IBM\Install\setup.exe" /w /z"sfc:\IBM\install\IEM_NT_Silent_Install.txt" /s /f2"c:\IBM\Install\Silent_NT.log"

During my download, I will extract the .TMP file and then copy the files to “C:\IBM\Install”. This is not required as they can be executed from the “__Download” folder, but I just wanted the code to be kept local. I am also going to create the response file called “IEM_NT_Silent_Install.txt” using the appendfile command.

Here is what my action script looks like
=======================================================================
prefetch NT_063000000_WINNT.tmp sha1:32685b95f970e8f61e187d8433360072f213dba8 size:85360201 http://winbfsrv:52311/Uploads/32685b95f970e8f61e187d8433360072f213dba8/NT_063000000_WINNT.tmp

extract NT_063000000_WINNT.tmp

dos mkdir "c:\IBM\install"
delete "__Download\NT_063000000_WINNT.tmp"

dos xcopy /s/e/v/q/y "__Download\*.*" "C:\IBM\Install"

parameter "TEMS_HOSTNAME"="temhub"
parameter "Install Directory"="C:\IBM\ITM"
parameter "FIREWALL"="NO"
parameter "NETWORK_PROTOCOL"="IP.PIPE"
parameter "IP_PIPE_PORT_NUMBER"="1918"

delete __appendfile
appendfile [INSTALLATION SECTION]
appendfile ;---------------------------------------------------------------------
appendfile AgentDeploy=yes
appendfile License Agreement=I agree to use the software only in accordance with the installed license.
appendfile Install Directory={parameter "Install Directory"}
appendfile Install Folder=IBM Tivoli Monitoring
appendfile EncryptionKey=IBMTivoliMonitoringEncryptionKey
appendfile [FEATURES]
appendfile KNTWICMA=Monitoring Agent for Windows OS

appendfile [CMA_CONFIG]
appendfile FTO Flag=N
appendfile AUDIT=Y
appendfile Protocol1={parameter "NETWORK_PROTOCOL"}
appendfile IPPIPE_Port={parameter "IP_PIPE_PORT_NUMBER"}
appendfile IPPIPE_Host={parameter "TEMS_HOSTNAME"}
appendfile CONNECT=Y
appendfile FIREWALL={parameter "FIREWALL"}

delete "c:\IBM\Install\IEM_NT_Silent_Install.txt"

copy __appendfile "c:\IBM\Install\IEM_NT_Silent_Install.txt"

waithidden "C:\IBM\Install\setup.exe" /w /z"sfc:\IBM\install\IEM_NT_Silent_Install.txt" /s /f2"c:\IBM\Install\Silent_NT.log"
=======================================================================

Now this action script does work and would be ok if you always had the same TEMS/RTEMS, but I am sure that is not realistic. If you want to add to it, I would look at possibly prompting for the TEMS/RTEMS host name with the “action parameter query”.

For example, you would replace the:
parameter "TEMS_HOSTNAME"="temhub"

with:
action parameter query "Primary RTEMS Hostname" with description "Please enter the Primary RTEMS Hostname:" with default value "ITMHUB"

Determine relevance method

There are many ways to check the relevance, this really comes down to your preferred method. Here are a couple options:
Files:
C:\IBM\ITM\INSTALL\ver\ KINWIINSMSTR.VER
C:\IBM\ITM\properties\version\knt.IBM_Tivoli_Monitoring_-_Windows_OS_Agent-6.3.0.swtag

Services:
KNTCMA_Primary
KNTCMA_Watchdog

For simplicity, I will just use the service “KNTCMA_Primary”. The relevance will look like:
not exists service “KNTCMA_Primary”

Create the fixlet

With all the information gathered and the TMP file copied to the Uploads folder, the fixlet can be created and used to deploy ITM agents. The fixlet can be created by putting the following into a text file with the BES extension and then use the Import function from the IEM Console to import the file to the desired site.

=================================================================
<?xml version="1.0" encoding="UTF-8"?>
<BES xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="BES.xsd">
        <Fixlet>
                <Title>Deploy: IBM Tivoli Monitoring Agent for Winnt</Title>
                <Description><![CDATA[Created: Jan 5, 2014<BR>Created By: Martin Carnegie, Gulf Breeze Software Partners<BR>Comment: Sample Fixlet for deploying ITM agent.]]></Description>
                <Relevance>(name of it contains "Win") of operating system </Relevance>
                <Relevance>not exists service "KNTCMA_Primary"</Relevance>
                <Category></Category>
                <Source>Internal</Source>
                <SourceID></SourceID>
                <SourceReleaseDate>2014-01-05</SourceReleaseDate>
                <SourceSeverity></SourceSeverity>
                <CVENames></CVENames>
                <SANSID></SANSID>
                <MIMEField>
                        <Name>x-fixlet-modification-time</Name>
                        <Value>Mon, 06 Jan 2014 04:57:48 +0000</Value>
                </MIMEField>
                <Domain>BESC</Domain>
                <DefaultAction ID="Action1">
                        <Description>
                                <PreLink>Click </PreLink>
                                <Link>here</Link>
                                <PostLink> to deploy this action.</PostLink>
                        </Description>
                        <ActionScript MIMEType="application/x-Fixlet-Windows-Shell">prefetch NT_063000000_WINNT.tmp sha1:32685b95f970e8f61e187d8433360072f213dba8 size:85360201 http://winbfsrv:52311/Uploads/32685b95f970e8f61e187d8433360072f213dba8/NT_063000000_WINNT.tmp

extract NT_063000000_WINNT.tmp

dos mkdir "c:\IBM\install"
delete "__Download\NT_063000000_WINNT.tmp"

dos xcopy /s/e/v/q/y "__Download\*.*" "C:\IBM\Install"
action parameter query "Primary RTEMS Hostname" with description "Please enter the Primary RTEMS Hostname:" with default value "ITMHUB"

parameter "TEMS_HOSTNAME"="temhub"
parameter "Install Directory"="C:\IBM\ITM"
parameter "FIREWALL"="NO"
parameter "NETWORK_PROTOCOL"="IP.PIPE"
parameter "IP_PIPE_PORT_NUMBER"="1918"

delete __appendfile
appendfile [INSTALLATION SECTION]
appendfile ;---------------------------------------------------------------------
appendfile AgentDeploy=yes
appendfile License Agreement=I agree to use the software only in accordance with the installed license.
appendfile Install Directory={parameter "Install Directory"}
appendfile Install Folder=IBM Tivoli Monitoring
appendfile EncryptionKey=IBMTivoliMonitoringEncryptionKey
appendfile [FEATURES]
appendfile KNTWICMA=Monitoring Agent for Windows OS

appendfile [CMA_CONFIG]
appendfile FTO Flag=N
appendfile AUDIT=Y
appendfile Protocol1={parameter "NETWORK_PROTOCOL"}
appendfile IPPIPE_Port={parameter "IP_PIPE_PORT_NUMBER"}
appendfile IPPIPE_Host={parameter "TEMS_HOSTNAME"}
appendfile CONNECT=Y
appendfile FIREWALL={parameter "FIREWALL"}

delete "c:\IBM\Install\IEM_NT_Silent_Install.txt"

copy __appendfile "c:\IBM\Install\IEM_NT_Silent_Install.txt"

waithidden "C:\IBM\Install\setup.exe" /w /z"sfc:\IBM\install\IEM_NT_Silent_Install.txt" /s /f2"c:\IBM\Install\Silent_NT.log"


</ActionScript>
                </DefaultAction>
        </Fixlet>
</BES>
=================================================================

After Thoughts

There are many ways that this could be done and some people may prefer to use the SPD/SPB file or even the IEM Software Distribution Wizards rather than this sort of manual method that I have here. Once you have the response files created and the command lines to use, the SPD file section will not be required.

I think it would also be interesting to use the “SWTAG” files “C:\IBM\ITM\properties\version” on Windows or the “VER” files in “C:\IBM\ITM\INSTALL\ver” as part of an analysis as there is some useful information in there to query the ITM environment. This would have to be explored more on the various agents to see where these files are and if they are there for all agents.

Hope you were able to get some useful information out of this. Please let me know if you come across an issue or a possible enhancement to this process.

Thanks


Martin Carnegie.