PDF - SCO 2.3.0.1 Troubleshooting VMControl Image Issues in Glance

ICO all-in-one deployment doesn't install Workload Deployer

The ICO 2.4 and 2.4.0.1 Deployment Server installers are definitely great pieces of software. They check prerequisites correctly and install without a hitch. They are really, really nice.

This post is about one of the options in the 'ds wizard' script options. Specifically, when you take the path:

[0] New IBM Cloud Orchestrator deployment.
[0] Demo
[1] All In One
[2] All In One IBM Cloud Orchestrator Deployment
   - Deploys IBM Cloud Orchestrator all-in-one in one machine.


This path actually deploys the template sco-devstack (/opt/ico_install/topology_templates/sco_devstack.json), which does NOT install the IBM Workload Deployer component. The effect that this has is that you do NOT have a PATTERNS option in the Administrator GUI (https://machine.fqdn:8443/), and so you can't create, view or use any patterns.

So if you want an all-in-one deployment, you should take a different option. Specifically, the path you should take through the wizard is:

[0] New IBM Cloud Orchestrator deployment.
[0] Demo
[1] All In One
[2] All In One IBM Cloud Orchestrator Deployment 
[0] All In One with addtional KVM compute node
   - Deploys IBM Cloud Orchestrator all-in-one KVM topology with an additional KVM compute node where components use a shared 
   - internal database.

And when it asks you for the FQDNs of the three servers, specify the FQDN of the ONE server that you'll be using in each of those three places.

A BIG note about running ICO 2.4.0.1

I successfully installed ICO 2.4.0.1 using 2 machines:

1: Deployment Server (where I could then run the 'ds wizard' command)

2: My all-in-one deployment to a KVM hypervisor.

Both machines are running RHEL 6.5 and have lots of CPU, RAM and disk.

My HUGE problem after install was that the Deployment Server automatically started an unnecessary dnsmasq process. And because of this, I could not create any new instances. Every time I tried, I got errors similar to the following in /var/log/nova/compute.log (and similar in network.log also):

2015-01-27 13:29:14.987 19521 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Remote error: ProcessExecutionError Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=3 dnsmasq --strict-order --bind-interfaces --conf-file=/etc/dnsmasq.conf --pid-file=/var/lib/nova/networks/nova-br1002.pid --listen-address=10.10.1.3 --except-interface=lo --dhcp-range=set:franknet,10.10.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br1002.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br1002.hosts
Exit code: 2
Stdout: u''
Stderr: u"2015-01-27 13:29:13.983 10307 INFO nova.openstack.common.periodic_task [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Skipping periodic task _periodic_update_dns because its interval is negative\n2015-01-27 13:29:13.985 10307 INFO nova.network.driver [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Loading network driver 'nova.network.linux_net'\n2015-01-27 13:29:13.986 10307 DEBUG nova.servicegroup.api [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.6/site-packages/nova/servicegroup/api.py:65\n2015-01-27 13:29:14.107 10307 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.6/site-packages/stevedore/extension.py:156\n2015-01-27 13:29:14.121 10307 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.6/site-packages/stevedore/extension.py:156\n\ndnsmasq: failed to create listening socket for 10.10.1.3: Address already in use\n"
[u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 193, in deallocate_for_instance\n    super(FloatingIP, self).deallocate_for_instance(context, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 563, in deallocate_for_instance\n    instance=instance)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 246, in deallocate_fixed_ip\n    address, instance=instance)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 980, in deallocate_fixed_ip\n    self._teardown_network_on_host(context, network)\n', u'  File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1906, in _teardown_network_on_host\n    self.driver.update_dhcp(elevated, dev, network)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1004, in update_dhcp\n    restart_dhcp(context, dev, network_ref)\n', u'  File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1118, in restart_dhcp\n    _execute(*cmd, run_as_root=True)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1211, in _execute\n    return utils.execute(*cmd, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute\n    return processutils.execute(*cmd, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 196, in execute\n    cmd=sanitized_cmd)\n', u'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=3 dnsmasq --strict-order --bind-interfaces --conf-file=/etc/dnsmasq.conf --pid-file=/var/lib/nova/networks/nova-br1002.pid --listen-address=10.10.1.3 --except-interface=lo --dhcp-range=set:franknet,10.10.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br1002.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br1002.hosts\nExit code: 2\nStdout: u\'\'\nStderr: u"2015-01-27 13:29:13.983 10307 INFO nova.openstack.common.periodic_task [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Skipping periodic task _periodic_update_dns because its interval is negative\\n2015-01-27 13:29:13.985 10307 INFO nova.network.driver [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Loading network driver \'nova.network.linux_net\'\\n2

Once I killed the offending dnsmasq process, I could successfully create instances.

SOURCE: http://fedoraproject.org/wiki/QA:Testcase_launch_an_instance_on_OpenStack

How to remove a bridged network interface in Red Hat Linux

So ICO 2.4 requires the Deployment Server machine to have interface br0 defined as a bridge connected to eth0. However, ICO 2.4.0.1 does NOT want br0, but wants eth0. I had already created br0 using the virsh command, so I had to look up how to remove it, and it's very simple once you find it:

virsh iface-unbridge br0

and that's it. It puts all of the configuration back on eth0 and you're in business.

I'm giving up on ICO 24 on RHEL 6.6

After installing the Deployment Service (which gives you the ds command), I tried to deploy an all-in-one ICO environment, and it failed very badly. So badly that multiple components stopped running and there was no recovering from it. So I'm now starting over with RHEL 6.5.

Getting IBM Cloud Orchestrator to install on Red Hat Enterprise Linux 6.6

This is probably fixed in ICO 2.4.1, but in 2.4, you need to modify a file to get the installation to work on RHEL 6.6 because it believes that only RHEL 6.4 and 6.5 are supported. You'll encounter an error when you run through the ds wizard command.

The file you need to update (after installing the Deployment Service) is:

/opt/ibm/cloud-deployer/precheck.json

In that file, you'll see multiple stanzas similar to this:

             "os": {
                        "release": "Red Hat Enterprise Linux Server release",
                        "arch": "64",
                        "version": ["6.5","6.4"]

               },

You need to update the line that contains "version" to be:

                        "version": ["6.6","6.5","6.4"]

And then the wizard can complete.

Some background:

I found this because I saw the error: 

"DeployTaskFailed: Failed to execute deployment task: deploy-precheck, error message: precheck failed on ico24demo.mynet.foo: ico24demo.mynet.fooincorrect os version,expected:['6.5', '6.4']actually:6.6\n\n"

And I ran the following command in multiple directories:

grep -r "6\.5" *

until I found the file I needed to change.

Configuring RHEL prerequisites for IBM Cloud Orchestrator 2.4

Introduction

The first basic requirement for your systems when deploying ICO 2.4 is to have DNS name resolution working correctly. For some reason or another, this is the area that has the largest number and most frequent problems in every single enterprise. So this short post is meant to help you get something going quickly, normally in a test environment, until you can communicate with your networking team to get the problems solved permanently.

My Simple Environment

My test environment is the demo configuration with one RHEL 6.6 server with 8 cores and 32GB of RAM, with KVM for virtualization.

Setting Your Hostname

You need to set your hostname to a Fully Qualified Domain Name (FQDN), and in a test environment, the format just needs to be correct - the actual name doesn't have to be registered anywhere. For example, I named my host ico24demo.mynet.foo. I don't care if .foo is a valid root or not because I get to make the rules in my own environment.

To permanently set your hostname to an FQDN if you didn't do it at install time, you need to edit the file:

/etc/sysconfig/network

and set:

HOSTNAME=your.full.fqdn

If you don't want to reboot to have it set, also run the following as root:

hostname your.full.fqdn

Update Your Hosts File

You also need to update your /etc/hosts file with your hostname and IP. So edit /etc/hosts and add that information for your host. In my environment, my hostname is ico24demo.mynet.foo and my static IP address is 192.168.1.250. So here's my /etc/hosts entry:

192.168.1.250     ico24demo.mynet.foo   ico24demo

Start dnsmasq

dnsmasq is a simple DNS server (and DHCP and some other things) that we'll configure locally for name resolution.

I picked a pretty standard installation for my Red Hat server (software development system with all of the Virtualization options selected), and it automatically installed the dnsmasq-2.48.14.el6.x86_64 package. You can check to see if you have this package installed with:

rpm -q -a | grep dnsmasq

If it's not installed, install it.

Assuming it is installed, make sure it's not already running with:

ps -ef | grep dnsmasq

If it is, kill it with:

service stop dnsmasq

The above may or may not work. If the process is still running kill it with the kill command.

And now that your /etc/hosts file has your host information, start dnsmasq with:

service start dnsmasq

Test Your New DNS Server

Probably the easiest way to test your server is to first edit your /etc/resolv.conf file to set your nameserver to your local machine. So my local IP is 192.168.1.250, and the IP address of my subnet's nameserver is 192.168.1.1. So in my /etc/resolv.conf file, I have the following:

# Generated by NetworkManager
nameserver 192.168.1.250
nameserver 192.168.1.1

Now run the command:

host 192.168.1.250

The correct output for me is:

250.1.168.192.in-addr.arpa domain name pointer ico24demo.mynet.foo.

If you get something different, go back over the steps above. But if it's correct, keep going.

Make the Nameserver Change Permanent

Go to System->Preferences->Network Connections to set your nameserver to be your local IP address.

A non-GUI way to do this is to update your /etc/sysconfig/network-scripts/ifcfg-eth0 file to set the DNS server.

Good luck!