Monday, March 2, 2015
IBM InterConnect Presentation - Application Performance Management and IT Service Management Best Practices
Thursday, January 29, 2015
The ICO 2.4 and 184.108.40.206 Deployment Server installers are definitely great pieces of software. They check prerequisites correctly and install without a hitch. They are really, really nice.
This post is about one of the options in the 'ds wizard' script options. Specifically, when you take the path:
 New IBM Cloud Orchestrator deployment.
 All In One
 All In One IBM Cloud Orchestrator Deployment
- Deploys IBM Cloud Orchestrator all-in-one in one machine.
This path actually deploys the template sco-devstack (/opt/ico_install/topology_templates/sco_devstack.json), which does NOT install the IBM Workload Deployer component. The effect that this has is that you do NOT have a PATTERNS option in the Administrator GUI (https://machine.fqdn:8443/), and so you can't create, view or use any patterns.
So if you want an all-in-one deployment, you should take a different option. Specifically, the path you should take through the wizard is:
 New IBM Cloud Orchestrator deployment.
 All In One
 All In One IBM Cloud Orchestrator Deployment
 All In One with addtional KVM compute node
- Deploys IBM Cloud Orchestrator all-in-one KVM topology with an additional KVM compute node where components use a shared
- internal database.
And when it asks you for the FQDNs of the three servers, specify the FQDN of the ONE server that you'll be using in each of those three places.
Tuesday, January 27, 2015
I successfully installed ICO 220.127.116.11 using 2 machines:
1: Deployment Server (where I could then run the 'ds wizard' command)
2: My all-in-one deployment to a KVM hypervisor.
Both machines are running RHEL 6.5 and have lots of CPU, RAM and disk.
My HUGE problem after install was that the Deployment Server automatically started an unnecessary dnsmasq process. And because of this, I could not create any new instances. Every time I tried, I got errors similar to the following in /var/log/nova/compute.log (and similar in network.log also):
2015-01-27 13:29:14.987 19521 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Remote error: ProcessExecutionError Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=3 dnsmasq --strict-order --bind-interfaces --conf-file=/etc/dnsmasq.conf --pid-file=/var/lib/nova/networks/nova-br1002.pid --listen-address=10.10.1.3 --except-interface=lo --dhcp-range=set:franknet,10.10.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br1002.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br1002.hosts
Exit code: 2
Stderr: u"2015-01-27 13:29:13.983 10307 INFO nova.openstack.common.periodic_task [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Skipping periodic task _periodic_update_dns because its interval is negative\n2015-01-27 13:29:13.985 10307 INFO nova.network.driver [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Loading network driver 'nova.network.linux_net'\n2015-01-27 13:29:13.986 10307 DEBUG nova.servicegroup.api [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.6/site-packages/nova/servicegroup/api.py:65\n2015-01-27 13:29:14.107 10307 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.6/site-packages/stevedore/extension.py:156\n2015-01-27 13:29:14.121 10307 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.6/site-packages/stevedore/extension.py:156\n\ndnsmasq: failed to create listening socket for 10.10.1.3: Address already in use\n"
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 193, in deallocate_for_instance\n super(FloatingIP, self).deallocate_for_instance(context, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 563, in deallocate_for_instance\n instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 246, in deallocate_fixed_ip\n address, instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 980, in deallocate_fixed_ip\n self._teardown_network_on_host(context, network)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1906, in _teardown_network_on_host\n self.driver.update_dhcp(elevated, dev, network)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1004, in update_dhcp\n restart_dhcp(context, dev, network_ref)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1118, in restart_dhcp\n _execute(*cmd, run_as_root=True)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1211, in _execute\n return utils.execute(*cmd, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute\n return processutils.execute(*cmd, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 196, in execute\n cmd=sanitized_cmd)\n', u'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=3 dnsmasq --strict-order --bind-interfaces --conf-file=/etc/dnsmasq.conf --pid-file=/var/lib/nova/networks/nova-br1002.pid --listen-address=10.10.1.3 --except-interface=lo --dhcp-range=set:franknet,10.10.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br1002.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br1002.hosts\nExit code: 2\nStdout: u\'\'\nStderr: u"2015-01-27 13:29:13.983 10307 INFO nova.openstack.common.periodic_task [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Skipping periodic task _periodic_update_dns because its interval is negative\\n2015-01-27 13:29:13.985 10307 INFO nova.network.driver [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Loading network driver \'nova.network.linux_net\'\\n2
Once I killed the offending dnsmasq process, I could successfully create instances.
Saturday, January 24, 2015
So ICO 2.4 requires the Deployment Server machine to have interface br0 defined as a bridge connected to eth0. However, ICO 18.104.22.168 does NOT want br0, but wants eth0. I had already created br0 using the virsh command, so I had to look up how to remove it, and it's very simple once you find it:
virsh iface-unbridge br0
and that's it. It puts all of the configuration back on eth0 and you're in business.
Friday, January 23, 2015
After installing the Deployment Service (which gives you the ds command), I tried to deploy an all-in-one ICO environment, and it failed very badly. So badly that multiple components stopped running and there was no recovering from it. So I'm now starting over with RHEL 6.5.
Thursday, January 22, 2015
This is probably fixed in ICO 2.4.1, but in 2.4, you need to modify a file to get the installation to work on RHEL 6.6 because it believes that only RHEL 6.4 and 6.5 are supported. You'll encounter an error when you run through the ds wizard command.
The file you need to update (after installing the Deployment Service) is:
In that file, you'll see multiple stanzas similar to this:
"release": "Red Hat Enterprise Linux Server release",
IntroductionThe first basic requirement for your systems when deploying ICO 2.4 is to have DNS name resolution working correctly. For some reason or another, this is the area that has the largest number and most frequent problems in every single enterprise. So this short post is meant to help you get something going quickly, normally in a test environment, until you can communicate with your networking team to get the problems solved permanently.
My Simple EnvironmentMy test environment is the demo configuration with one RHEL 6.6 server with 8 cores and 32GB of RAM, with KVM for virtualization.
Setting Your HostnameYou need to set your hostname to a Fully Qualified Domain Name (FQDN), and in a test environment, the format just needs to be correct - the actual name doesn't have to be registered anywhere. For example, I named my host ico24demo.mynet.foo. I don't care if .foo is a valid root or not because I get to make the rules in my own environment.
To permanently set your hostname to an FQDN if you didn't do it at install time, you need to edit the file:
If you don't want to reboot to have it set, also run the following as root:
Update Your Hosts FileYou also need to update your /etc/hosts file with your hostname and IP. So edit /etc/hosts and add that information for your host. In my environment, my hostname is ico24demo.mynet.foo and my static IP address is 192.168.1.250. So here's my /etc/hosts entry:
192.168.1.250 ico24demo.mynet.foo ico24demo
rpm -q -a | grep dnsmasq
If it's not installed, install it.
Assuming it is installed, make sure it's not already running with:
ps -ef | grep dnsmasq
If it is, kill it with:
service stop dnsmasq
The above may or may not work. If the process is still running kill it with the kill command.
And now that your /etc/hosts file has your host information, start dnsmasq with:
service start dnsmasq
Test Your New DNS ServerProbably the easiest way to test your server is to first edit your /etc/resolv.conf file to set your nameserver to your local machine. So my local IP is 192.168.1.250, and the IP address of my subnet's nameserver is 192.168.1.1. So in my /etc/resolv.conf file, I have the following:
# Generated by NetworkManager
The correct output for me is:
250.1.168.192.in-addr.arpa domain name pointer ico24demo.mynet.foo.
If you get something different, go back over the steps above. But if it's correct, keep going.
Make the Nameserver Change PermanentGo to System->Preferences->Network Connections to set your nameserver to be your local IP address.
A non-GUI way to do this is to update your /etc/sysconfig/network-scripts/ifcfg-eth0 file to set the DNS server.
Monday, July 7, 2014
If you've installed APM UI v7.7 on a Windows server, you probably have noticed that IBM doesn't create Windows services. As a result, none of the services start after a reboot, etc.
Below are the steps necessary to create Windows services for APM UI and SCR (Service Component Repository). In this scenario, we have the SCR database using DB2 vs. Derby.
We've also included the commands necessary to create the service for SCR on Derby (however, this hasn't been tested).
Our base install directory for the APM UI is C:\IBM\APMUI - your path may differ, adjust the commands below as appropriate.
- Download the Apache Commons Daemon (link).
- Extract commons-daemon-1.0.15-bin-windows.zip and copy prunsrv.exe to C:\IBM\APMUI\bin\
- Open a Command Prompt and change into your C:\IBM\APMUI\bin directory
- Create the Windows service for SCR and have it depend on Tcpip and DB2 being up first.
prunsrv //IS//SCR --Startup=auto --DisplayName="IBM APMUI - SCR" --Description="IBM WebSphere Liberty Profile SCR" ++DependsOn=Tcpip ++DependsOn=DB2-0 --LogPath=C:\IBM\APMUI\usr\servers\apmui\logs --StdOutput=auto --StdError=auto --StartMode=exe --StartPath=C:\IBM\APMUI --StartImage=C:\IBM\APMUI\bin\server.bat --StartParams=start#scr --StopMode=exe --StopPath=C:\IBM\APMUI --StopImage=C:\IBM\APMUI\bin\server.bat --StopParams=stop#scr
- Create the Windows service for APMUI and have it depend on SCR being up first.
prunsrv //IS//APMUI --Startup=auto --DisplayName="IBM APMUI - APMUI" --Description="IBM WebSphere Liberty Profile APMUI" ++DependsOn=SCR --LogPath=C:\IBM\APMUI\usr\servers\apmui\logs --StdOutput=auto --StdError=auto --StartMode=exe --StartPath=C:\IBM\APMUI --StartImage=C:\IBM\APMUI\bin\server.bat --StartParams=start#apmui --StopMode=exe --StopPath=C:\IBM\APMUI --StopImage=C:\IBM\APMUI\bin\server.bat --StopParams=stop#apmui
Derby vs. DB2If you have SCR running under Derby vs. DB2, you can create a third service for SCRDERBY. The start-up of SCR would then depend on SCRDERBY instead of DB2-0.
- Create the Windows service for SCRDERBY and have it depend on TCPIP being up first
- In this case, have the SCR service depend on SCRDERBY instead of DB2.
The name specified after //IS// is effectively the "short name" for the service. So, you can issue commands such as "net start apmui", "net stop scrderby", etc. based on those names. As with all services, you can use the full Display Name if you enjoy typing (net start "IBM APMUI - APMUI", etc.)
The typical startup sequence would be:
- net start scrderby (if running Derby vs. DB2)
- net start scr
- net start apmui
The typical shutdown sequence would be:
- net stop apmui
- net stop scr
- net stop scrderby (if running Derby vs. DB2)
prunsrv Documentation/Usage:Here's a link to the prunsrv documentation/usage.
Sample - APMUI Service
Wednesday, January 29, 2014
MongoDB Setup Overview
- Create 3 databases, each configured to use unique paths and ports (ports: 37017, 37018, 37019)
- Connect to the "soon-to-be" primary database (port 37017) and configure the replica set.
- Add second (port 37018) and third (port 37019) databases to the replica set.
- Installed a demo database using JSON import
- Started a Configuration Server (port 37020)
- Started mongos, pointing at Configuration Server and listening on port 37021
- Connected to mongos (port 37021)
- Added replica set to Shard
- Enabled Sharding on our demodb
- Enabled Sharding on the collection (aka table)
A special thanks to the following website for providing some demo database data and a nice presentation on MongoDB and Ruby apps. Also thanks to the MongoDB docs for creating a cluster.
You may ask, why 3 databases, aren't 2 enough? This is because the recommended MINIMUM number of databases for a Replica Set is 3. If your setup only has 2 databases, Application Insight will flag the Global MongoDB Status as a Warning.
Replica Set is named "rs1".
Download MongoDB (we used 64-bit Linux) from here.
Untar file. Location of files will be referred to as $MEDIA
The remaining commands assume the $MEDIA/bin directory are in your path.
mkdir -p /srv/mongodb/rs1-0 /srv/mongodb/rs1-1 /srv/mongodb/rs1-2
# Start up 3 databases
# Connect to to-be-primary
#Give Primary a configuration for a replication set
#Initiates the replica set
#Displays Replication set
#Add 2nd db - NOTE: you might need to wait for prompt to change to PRIMARY before continuing.
#Add 3rd db
#Display Replication set, confirm three are listed
#Start a config server
#Start mongos pointing at config server
#Connect to mongos
#Add replica set to shard
#Shows replica set and other stats
#Connect to mongos
#Enable sharding for the DB
#Set index for sharding
#Setup sharding of the collection (table)
MongoDB Agent Setup
mongodb-agent.sh config pulse21
Application Insight Screenshots
Anthony Segelhorst and Jamie Carl
Monday, January 6, 2014
- Create IEM Download File
- Determine the sha1 information.
- Review the SPD file and determine install command and options for the NT_Silent_Install.txt
- Use the BFArchive tool to ZIP the bundle directory and upload to the IEM server.
- Determine relevance method
- Create the fixlet
Create the Bundle
Create IEM Download File
Determine sha1 information
Review the SPD file and determine install command and options for the NT_Silent_Install.txt
Determine relevance method
Create the fixlet
Wednesday, December 18, 2013
If you currently using APM UI 7.6.x on mobile devices, I would currently recommend keeping the APM UI 7.6.x around and running. Keeping both running should not be an issue, since APM UI 7.6 uses TIP and APM UI 7.7 uses Liberty.
When testing APM UI 7.7 on an IPAD running IOS 7.0.4 the user interface was not usable. I saw the following issues:
1. Everything was not visible on the screen right to left on IBM provided widgets. This was even after collapsing the left pane.
2. I could not scroll down to see the data on all widgets.
Based on the following Dev Works post from IBM, I believe true mobile support is not expected till March.
NOTE: We also found the following in the manuals:
To avoid conflicts with the SCR of Tivoli Business Service Manager, do not install SmartCloud Application Performance Management UI and Tivoli Business Service Manager on the same computer.
This is probably due to the fact the TBSM schema and APM UI are very similar.
Source --> http://publib.boulder.ibm.com/infocenter/tivihelp/v63r1/topic/com.ibm.apm.doc_7.7/apm_ui_docs/apmui_77/install_using_im.html
Sunday, December 15, 2013
APMUI 7.7 is GA
- Windows - CIR57ML
- AIX - CIR58ML
- Linux X - CIR59ML
- Linux Z - CIR5AML
Tuesday, November 5, 2013
One of the issues that I have seen on customer sites are problems with WMI (Windows Management Instrumentation). One of the (many) good things about IEM is that it really does not need WMI to function as IBM uses custom built inspectors to do almost everything. The only issues are mainly around some data collection points (some hardware information for example), but this does not cause the agent to fail and still allows for actions to be taken on the targets. This is a huge advantage for IEM, as other vendor products will not even function if WMI is broken.
To identify systems with WMI issues is pretty simple with a Web Report:
1. In Web Reports go to Explore Data
2. Click the Edit Columns and add the column "Computer Model - Windows" (part of the hardware analysis)
3. Add the filter "Computer" "Computer - Model - Windows (Hardware Information (Windows))" "contains" "Windows Error"
This will show all systems with errors which should be ones with WMI issues. The thing to note is that there are different error codes and some are easier to fix than others.
Now to repair the WMI issues, I created a fixlet that can be used. This seems to address some of the errors that I have seen, but not all. Some systems are so far gone, that it will require manual intervention or even a rebuild of the OS.
I have made this available on the BigFix.me site for downloading and comments:
Please note that this has been tested but only in a limited amount as I do not have many systems available to me and I could not simulate all the possible WMI errors.
Let me know how this works for you and we will see if we can get this to fix all errors.
Tuesday, October 29, 2013
This has almost nothing to do with Tivoli, but I hit this stupid problem every few years, and it takes me 30 minutes to figure out the solution each time. So I thought I would post it here:
When creating a fat GUI in Java, you may have a portion of your window that is a JScrollPane, which has a ViewPort and a ViewPortView (often a JPanel) . A common need is to add components to your JPanel (the ViewPortView). However, when you do this, you'll notice that whatever you do with the JScrollPane's scrollbars, it won't scroll in one or both directions. Well, the problem is that the addition of new components does NOT increase the value of the ViewPortView's PreferredSize attribute. So as you add more components, you need to manually update this attribute with code similar to:
I've tried setViewSize(), and that doesn't seem to do anything - setPreferredSize is what you've got to do.
Wednesday, October 23, 2013
After installing JazzSM 22.214.171.124, I learned AGAIN all browsers are not created equal.
The browser, version, and java version are becoming more and more important today within the IBM product set and here is another example.
So after having a completed JazzSM/TCR install I went to configure the DB connection for TDW via Launch > Administration and found the following screen appears. Where is the rest of it?
The troubleshooting starts:
1. Maybe the installer said the install was SUCCESS, but it really failed. Better check the logs.
2. Maybe I need to cycled the application.
3. Let me check the user permissions.
4. Lets create another user.
5. Did the Cognos content store DB get created.
After talking to Venkat Saranathan (the GBS resident TCR expert), he suggests trying I.E (his favorite browser) instead of Chrome.
If you use Firefox or I.E the gui draws correctly or at least the buttons are now there. I.E. had other issues since I was using version 10 and version 10 is not supported for TCR. So after trying Firefox I was able to complete the configuration.
If you want to see what browsers and version are supported for JazzSM 1.10.1 check out the following link:
--> Then Select prerequisites
Wednesday, October 16, 2013
Mark Poulson gave a great presentation on our company and what this achievement means. Click here to view the PDF of that presentation.