Showing posts with label ICO. Show all posts
Showing posts with label ICO. Show all posts
Tuesday, November 10, 2015
ICO 2.5: Azure deployment
Here's a video I created going over some of the details of deploying through IBM Cloud Orchestrator 2.5 to the Microsoft Azure cloud.
Monday, November 9, 2015
Using a certificate with an Azure deployment in ICO 2.5
The ICO 2.5 documentation clearly states that you cannot create a service that uses certificates from ICO, and that is still true. HOWEVER, you can deploy into an EXISTING service that has a certificate associated. The trick here is that you have to first manually create the service in Azure, then manually attach the certificate in Azure. Then when you select the "Deploy cloud service into Azure region" Self-Service option, select the "Use an existing Cloud Service" option.
Friday, November 6, 2015
ICO 2.5: Deploy LAMP stack additional requirements
Once you get ICO 2.5 up and running, you'll have a few self-service offerings available in the dashboard, including "Deploy LAMP stack". This offering requires just a couple of additional configuration steps in addition to the documentation to get it to completely work.
1. You MUST have a flavor named "m1.medium" in your project and it must be large enough for your image named "linux_img".
2. Your linux_img needs to have the yum repositories available that provide httpd, mysql and php. On CentOS and RHEL, these packages are on the base DVD. To provide these to the VMs on my private network, I have these DVDs mounted under a central (private) web server's web root, and have these URLs configured in the appropriate /etc/yum.repos.d/*.repo files on my linux_img image.
3. The MySQL expects to have access to the file named:
/tmp/mysqlinstaller/world.sql
It runs this SQL script during install. You can download this file from
http://downloads.mysql.com/docs/world.sql.gz
I simply placed this file on my image so it's there for every VM created.
That's it.
1. You MUST have a flavor named "m1.medium" in your project and it must be large enough for your image named "linux_img".
2. Your linux_img needs to have the yum repositories available that provide httpd, mysql and php. On CentOS and RHEL, these packages are on the base DVD. To provide these to the VMs on my private network, I have these DVDs mounted under a central (private) web server's web root, and have these URLs configured in the appropriate /etc/yum.repos.d/*.repo files on my linux_img image.
3. The MySQL expects to have access to the file named:
/tmp/mysqlinstaller/world.sql
It runs this SQL script during install. You can download this file from
http://downloads.mysql.com/docs/world.sql.gz
I simply placed this file on my image so it's there for every VM created.
That's it.
ICO 2.5: Where does the heat template for the "Deploy LAMP stack" come from?
ICO 2.5 comes with a few samples in the Self-Service Catalog, including one named "Deploy LAMP stack" (under "Deploy customized cloud services"). Once you have your linux_img image configured correctly and your linux_key key pair created, you can request this offering to deploy a LAMP stack among three machines: Apache, MySQL and PHP. I found the heat template command logs on each machine in /var/log/cloud-init-output.log and /tmp/install.log, and wanted to find where exactly that template was coming from. It took me far too long to find it because it's an inline string in the JavaScript code that can be seen with Business Process Designer. Specifically, you can find it in the Embeddable
Deploy Lamp Template service in the SCOrchestrator OpenStack Services toolkit,
in the Build Parameters element on the diagram, here:
You can see it in the green text as part of the "Implementation" in the bottom panel.
You can see it in the green text as part of the "Implementation" in the bottom panel.
ICO 2.5: Creating a Red Hat 6.5 image with Gnome for use with a VMWare vSphere cloud
Introduction
When creating your private OpenStack-managed vSphere cloud, you're going to need some "images" ("VM Templates" in VMWare terminology) so you can launch/deploy instances. The really sticky part about this configuration is that OpenStack has traditionally only supported the KVM hypervisor, which uses a different disk format than VMWare (KVM uses QCOW2 and VMWare uses VMDK). I found some great QCOW2 images here: http://docs.openstack.org/image-guide/content/ch_obtaining_images.html and some great CentOS VMDK images here: http://osboxes.org. I had some different hurdles with each of those and finally decided just to install RHEL 6.5 from scratch, then modify that VM to work, then create a VM Template that would be automatically discovered by OpenStack as an image. In this post I'll cover the highlights of this technique.Install Red Hat
Install the "Desktop" option. This will mean that you have to do a little cleanup later, but I'll cover that.
Install VMWare Tools
Go ahead and install VMWare tools just to make your life a little easier.
Configure Yum Repositories and Install Packages
If you're using Red Hat Subscription Manager, you don't have to go through these steps. If you're not using RHSN, you'll need to manually configure some repositories. I found for this exercise that the CentOS repositories, along with the EPEL repository, worked great. To configure the CentOS repositories, create a file named /etc/yum.repos.d/YOU_PICK_A_NAME.repo with the following contents:
[base]
name=CentOS-6 -
Base
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os
#baseurl=http://mirror.centos.org/centos/6/os/x86_64/
gpgcheck=0
#released updates
[updates]
name=CentOS-6 -
Updates
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=updates
#baseurl=http://mirror.centos.org/centos/6/updates/x86_64/
gpgcheck=0
#additional
packages that may be useful
[extras]
name=CentOS-6 -
Extras
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=extras
#baseurl=http://mirror.centos.org/centos/6/extras/x86_64/
gpgcheck=0
#additional
packages that extend functionality of existing packages
[centosplus]
name=CentOS-6 -
Plus
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=centosplus
#baseurl=http://mirror.centos.org/centos/6/centosplus/x86_64/
gpgcheck=0
enabled=0
#contrib - packages
by Centos Users
[contrib]
name=CentOS-6 -
Contrib
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=contrib
#baseurl=http://mirror.centos.org/centos/6/contrib/x86_64/
gpgcheck=0
enabled=0
Then run 'yum repo-list' to verify the repositories were created. Now you can run the following commands:
yum -y epel-release
yum -y cloud-init
yum -y cloud-utils
yum -y
heat-cfntools
Configure Networking
Since the "Desktop" option was chosen, the NetworkManager service was installed and enabled. You need to change that with these commands:
service
NetworkManager stop
chkconfig
NetworkManager off
service network start
chkconfig network
chkconfig network
Now run the following commands so that eth0 will get configured properly:
sed -i '/^HWADDR/d'
/etc/sysconfig/network-scripts/ifcfg-eth0
echo -n >
/etc/udev/rules.d/70-persistent-net.rules
echo -n > /lib/udev/rules.d/75-persistent-net-generator.rules
Configure cloud.cfg
The default configuration will try to contact multiple non-existent IP addresses for metadata, and you don't need that by changing /etc/cloud/cloud.cfg. Add the following line to that file:
datastore_list: [ NoCloud, ConfigDrive, None ]
You *may* also want to set:
disable_root: 0
ssh_pwauth: 1
This will allow you to access the VM as root, and via ssh with a password.
Make a Template
Via the vSphere Web Client or vSphere Windows Client, clone the VM to a template. After you do this, wait a few minutes for OpenStack to "see" this new template as an Image.
Conclusion
Following the above steps should get you a working template from which you can explore the other capabilities of ICO and ICMWO.
Monday, September 28, 2015
There's an updated cloud in town Part 3: Reinstalling everything
The prerequisite checker doesn't check the ICMWO fixpack
ICO 2.5 has a great prerequisite check script, but I found out the hard way that it doesn't check to make sure you have the correct ICMWO fixpack installed (fp 2 or later is required). Specifically, what I got was a crash of the script, rather than anything useful. Even after I made it so the script didn't crash, the error still didn't directly point to the fact that I had forgotten to install the fixpack. Here's the failure message I received at the end of the prereq_checker.sh output:
Traceback (most recent call last):
File "/mnt/hgfs/ico25/ico_install/installer/read-params.py", line 347, in <module>
if not validator.validate(param_dict, prereq, log_file) and not os.path.exists("/tmp/skip-checking"):
File "/mnt/hgfs/ico25/ico_install/installer/validator/__init__.py", line 74, in validate
openstackServices.validate(params, response1, messageValidator)
File "/mnt/hgfs/ico25/ico_install/installer/validator/openstackServices.py", line 70, in validate
if not user_response:
UnboundLocalError: local variable 'user_response' referenced before assignment
I edited line 70 of openstackServices.py to allow it to at least complete correctly, and this was the error message at that point:
- Checking that the following services are available: ['cinder', 'glance', 'nova', 'neutron|nova network']
- Status: Failed
- The following services must be available and running : ['cinder', 'glance', 'nova', 'neutron|nova network']
ERROR: The cinder service API is not enabled
- User response: Ensure that the openstack services are available
All of those services were running, but it jarred me into remembering that I hadn't installed FP3. Once I installed the fixpack, the prereq checker script and install were able to complete successfully.
Verifying you ran ICM_config_ico.sh correctly
To verify that you ran the ICM_config_ico.sh script correctly on your servers, open the OpenStack Dashboard (https://openstack_controller) and navigate to IDENTITY->Domains, then click the down arrow far to the right of a domain and select Edit Domain to get this dialog:Specifically, notice that there are 5 tabs: Domain Information, Domain Members, Domain Groups, Domain Administrators, Quota and Availability Zones. This is how it should look. If it doesn't, you did something wrong in running the ICM_config_ico.sh script. I think my problem was that I ran it twice on my controller - once with the "master_controller" option and once with the "controller" option. I should have only run it ONCE, with the "master_controller" option.
You will need to go to the Availability Zones tab and add the "nova" availability zone to the Default domain.
Also navigate to IDENTITY->Projects and click the down arrow to the far right of the "admin" project, then select Edit project and you should see this dialog:
Notice that it has an Availability Zones tab. Again, that is not there if the scripts didn't run correctly.
And now for the project (similar to the domain above), you will need to go to the Availability Zones tab and add the "nova" availability zone to the projects you'll be working with through ICO.
Thursday, September 24, 2015
There's an updated cloud in town Part 2: Still Installing ICO 2.5
A few more hurdles overcome as I get closer to getting ICO 2.5 installed.
systemctl stop firewalld
systemctl disable firewalld
In my case, it was blocking port 53 (dns), which I needed open to configure the vCenter server (next section). I first just used Applications->Sundry->Firewall to open port 53, then realized that I could just turn it completely off in my test environment so I don't hit any more problems with it.
I didn't want to get a Windows machine involved if at all possible, and it turns out to be fairly straightforward to do this. You will find 99% of the instructions in this great article:
http://www.unixarena.com/2015/05/how-to-deploy-vcsa-6-0-on-vmware-workstation.html
Specifically, the .OVA file can be found in the .ISO file that you download from VMWare. It just doesn't have a .ova extension. So you need to extract the file, change the name to include the .ova extension, and then you're mainly off to the races. HOWEVER, you have to do ONE MORE THING to actually get it working. Specifically, you need to add this additional line to the end of the .vmx file after you import the .ova file:
guestinfo.cis.appliance.net.dns.servers="172.16.30.8"
Some RHEL 7 notes
The firewall in RHEL7 (and 7.1) is not iptables. Instead, it's the firewalld service that's controlled by systemd. I'm not sure which install option causes it to be configured because it wasn't running on all of my RHEL 7.1 systems. Anyway, to turn it off, you can run:systemctl stop firewalld
systemctl disable firewalld
In my case, it was blocking port 53 (dns), which I needed open to configure the vCenter server (next section). I first just used Applications->Sundry->Firewall to open port 53, then realized that I could just turn it completely off in my test environment so I don't hit any more problems with it.
Installing vSphere 6.0 Without a Windows Machine
I decided to also install vSphere 6.0 to use that as a testbed, and that has a few challenges. Specifically, the vCenter Server Appliance (VCSA) no longer ships directly as a .OVA file. It is now an ISO file that you're supposed to mount and run on a Windows machine to remotely install the vCenter Server Appliance on a remote ESXi server.I didn't want to get a Windows machine involved if at all possible, and it turns out to be fairly straightforward to do this. You will find 99% of the instructions in this great article:
http://www.unixarena.com/2015/05/how-to-deploy-vcsa-6-0-on-vmware-workstation.html
Specifically, the .OVA file can be found in the .ISO file that you download from VMWare. It just doesn't have a .ova extension. So you need to extract the file, change the name to include the .ova extension, and then you're mainly off to the races. HOWEVER, you have to do ONE MORE THING to actually get it working. Specifically, you need to add this additional line to the end of the .vmx file after you import the .ova file:
guestinfo.cis.appliance.net.dns.servers="172.16.30.8"
Set the value appropriately for your network. If you don't add this, the VM will start up, but will have the error:
Failed to configure network
And I couldn't find a way to fix that in the VM as it stood. I updated the DNS settings, rebooted the server, did lots of other things, etc., and it still just showed that error. So I knew I would have to recreate the VM from the OVA file, but needed to figure out how to set the DNS server of the VM from the VMX file.
So I mounted the VCSA ISO file on Linux and ran the following command at the root of it:
grep -r guestinfo.cis *
Somewhat amazingly, that came back within seconds and I found all of the settings from the linked article, and then I searched for "dns" and found the above REQUIRED setting.
And I couldn't find a way to fix that in the VM as it stood. I updated the DNS settings, rebooted the server, did lots of other things, etc., and it still just showed that error. So I knew I would have to recreate the VM from the OVA file, but needed to figure out how to set the DNS server of the VM from the VMX file.
So I mounted the VCSA ISO file on Linux and ran the following command at the root of it:
grep -r guestinfo.cis *
Somewhat amazingly, that came back within seconds and I found all of the settings from the linked article, and then I searched for "dns" and found the above REQUIRED setting.
I didn't have a "good" DNS server on my network, so I quickly created a DNS server on one of my RHEL7.1 systems. It's REALLY easy to do this if you have all of your hosts in the /etc/hosts file. You just need to run the command:
service dnsmasq start
systemctl start dnsmasq
(Edit 9/26: I changed the above command to use the systemd mechanism for starting the service)
And that's it. You now have a DNS server.
I thought that the vSphere Web Client would allow me to just use a browser, but that's not quite right. the web interface requires Flash, and really only supports Windows or MacOS clients. So I've had to bring a Windows machine into the mix anyway.
systemctl start dnsmasq
(Edit 9/26: I changed the above command to use the systemd mechanism for starting the service)
And that's it. You now have a DNS server.
You still NEED a Windows machine for vSphere
I thought that the vSphere Web Client would allow me to just use a browser, but that's not quite right. the web interface requires Flash, and really only supports Windows or MacOS clients. So I've had to bring a Windows machine into the mix anyway.
Wednesday, September 23, 2015
There's an updated cloud in town Part 1: Installing ICO 2.5
IBM Cloud Orchestrator 2.5 was recently released, so I'm installing it in my lab. This new release is based on IBM Cloud Manager with OpenStack 4.3, so quite a bit has changed. This series of posts will discuss some of the issues with installation and overall thoughts as I go through.
If your target machines don't have access to the Red Hat Subscription Network, you can get around this by downloading Fedora 19 and add it as a Yum repository. But the Fedora ISO doesn't include the python-pyasn1-modules rpm. So you'll need to download that and add it to a local repository on the ICM Deployer machine. I used the 'createrepo' command to create the repository under /opt/ibm/cmwo/yum-repo/operatingsystem/redhat7.1/x86_64/optional .
I downloaded the Fedora 19 ISO from here: http://www.itsprite.com/free-linux-download-fedora-19-iso-cddvd-images/
And I downloaded the python-pyasn1-modules RPM from here: ftp://ftp.icm.edu.pl/vol/rzm5/linux-fedora-secondary/releases/19/Everything/armhfp/os/Packages/p/python-pyasn1-modules-0.1.6-1.fc19.noarch.rpm I think a better link is: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/19/Everything/x86_64/os/Packages/p/python-pyasn1-modules-0.1.6-1.fc19.noarch.rpm
I didn't need any other packages - the Fedora 19 ISO and the python-pyasn1-modules RPM gave my install everything it needed.
Wrong Version of IBM Cloud Manager
IBM Cloud Manager is included in the eAssembly for ICO, but it's not the required level! The Cloud Manager files included with ICO are JUST version 4.3. However, version 4.3 with at least fixpack 2 is what's required. So after installing Cloud Manager, you will need to go to IBM Fix Central to download Cloud Manager 4.3 fixpack 3, which was released after ICO 2.5.Cloud Manager: Additional YUM Repository Needed for KVM
If you're deploying a KVM cloud to Red Hat 7 or 7.1, you will need to enable the Red Hat "Optional" repository to have access to several python packages, including python-zope-interface and python-pyasn1-modules.If your target machines don't have access to the Red Hat Subscription Network, you can get around this by downloading Fedora 19 and add it as a Yum repository. But the Fedora ISO doesn't include the python-pyasn1-modules rpm. So you'll need to download that and add it to a local repository on the ICM Deployer machine. I used the 'createrepo' command to create the repository under /opt/ibm/cmwo/yum-repo/operatingsystem/redhat7.1/x86_64/optional .
I downloaded the Fedora 19 ISO from here: http://www.itsprite.com/free-linux-download-fedora-19-iso-cddvd-images/
And I downloaded the python-pyasn1-modules RPM from here: ftp://ftp.icm.edu.pl/vol/rzm5/linux-fedora-secondary/releases/19/Everything/armhfp/os/Packages/p/python-pyasn1-modules-0.1.6-1.fc19.noarch.rpm I think a better link is: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/19/Everything/x86_64/os/Packages/p/python-pyasn1-modules-0.1.6-1.fc19.noarch.rpm
I didn't need any other packages - the Fedora 19 ISO and the python-pyasn1-modules RPM gave my install everything it needed.
Thursday, January 29, 2015
ICO all-in-one deployment doesn't install Workload Deployer
The ICO 2.4 and 2.4.0.1 Deployment Server installers are definitely great pieces of software. They check prerequisites correctly and install without a hitch. They are really, really nice.
This post is about one of the options in the 'ds wizard' script options. Specifically, when you take the path:
[0] New IBM Cloud Orchestrator deployment.
[0] Demo
[1] All In One
[2] All In One IBM Cloud Orchestrator Deployment
- Deploys IBM Cloud Orchestrator all-in-one in one machine.
This path actually deploys the template sco-devstack (/opt/ico_install/topology_templates/sco_devstack.json), which does NOT install the IBM Workload Deployer component. The effect that this has is that you do NOT have a PATTERNS option in the Administrator GUI (https://machine.fqdn:8443/), and so you can't create, view or use any patterns.
So if you want an all-in-one deployment, you should take a different option. Specifically, the path you should take through the wizard is:
[0] New IBM Cloud Orchestrator deployment.
[0] Demo
[1] All In One
[2] All In One IBM Cloud Orchestrator Deployment
[0] All In One with addtional KVM compute node
- Deploys IBM Cloud Orchestrator all-in-one KVM topology with an additional KVM compute node where components use a shared
- internal database.
And when it asks you for the FQDNs of the three servers, specify the FQDN of the ONE server that you'll be using in each of those three places.
This post is about one of the options in the 'ds wizard' script options. Specifically, when you take the path:
[0] New IBM Cloud Orchestrator deployment.
[0] Demo
[1] All In One
[2] All In One IBM Cloud Orchestrator Deployment
- Deploys IBM Cloud Orchestrator all-in-one in one machine.
This path actually deploys the template sco-devstack (/opt/ico_install/topology_templates/sco_devstack.json), which does NOT install the IBM Workload Deployer component. The effect that this has is that you do NOT have a PATTERNS option in the Administrator GUI (https://machine.fqdn:8443/), and so you can't create, view or use any patterns.
So if you want an all-in-one deployment, you should take a different option. Specifically, the path you should take through the wizard is:
[0] New IBM Cloud Orchestrator deployment.
[0] Demo
[1] All In One
[2] All In One IBM Cloud Orchestrator Deployment
[0] All In One with addtional KVM compute node
- Deploys IBM Cloud Orchestrator all-in-one KVM topology with an additional KVM compute node where components use a shared
- internal database.
And when it asks you for the FQDNs of the three servers, specify the FQDN of the ONE server that you'll be using in each of those three places.
Tuesday, January 27, 2015
A BIG note about running ICO 2.4.0.1
I successfully installed ICO 2.4.0.1 using 2 machines:
1: Deployment Server (where I could then run the 'ds wizard' command)
2: My all-in-one deployment to a KVM hypervisor.
Both machines are running RHEL 6.5 and have lots of CPU, RAM and disk.
My HUGE problem after install was that the Deployment Server automatically started an unnecessary dnsmasq process. And because of this, I could not create any new instances. Every time I tried, I got errors similar to the following in /var/log/nova/compute.log (and similar in network.log also):
2015-01-27 13:29:14.987 19521 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Remote error: ProcessExecutionError Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=3 dnsmasq --strict-order --bind-interfaces --conf-file=/etc/dnsmasq.conf --pid-file=/var/lib/nova/networks/nova-br1002.pid --listen-address=10.10.1.3 --except-interface=lo --dhcp-range=set:franknet,10.10.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br1002.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br1002.hosts
Exit code: 2
Stdout: u''
Stderr: u"2015-01-27 13:29:13.983 10307 INFO nova.openstack.common.periodic_task [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Skipping periodic task _periodic_update_dns because its interval is negative\n2015-01-27 13:29:13.985 10307 INFO nova.network.driver [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Loading network driver 'nova.network.linux_net'\n2015-01-27 13:29:13.986 10307 DEBUG nova.servicegroup.api [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.6/site-packages/nova/servicegroup/api.py:65\n2015-01-27 13:29:14.107 10307 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.6/site-packages/stevedore/extension.py:156\n2015-01-27 13:29:14.121 10307 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.6/site-packages/stevedore/extension.py:156\n\ndnsmasq: failed to create listening socket for 10.10.1.3: Address already in use\n"
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 193, in deallocate_for_instance\n super(FloatingIP, self).deallocate_for_instance(context, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 563, in deallocate_for_instance\n instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 246, in deallocate_fixed_ip\n address, instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 980, in deallocate_fixed_ip\n self._teardown_network_on_host(context, network)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1906, in _teardown_network_on_host\n self.driver.update_dhcp(elevated, dev, network)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1004, in update_dhcp\n restart_dhcp(context, dev, network_ref)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1118, in restart_dhcp\n _execute(*cmd, run_as_root=True)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1211, in _execute\n return utils.execute(*cmd, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute\n return processutils.execute(*cmd, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 196, in execute\n cmd=sanitized_cmd)\n', u'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=3 dnsmasq --strict-order --bind-interfaces --conf-file=/etc/dnsmasq.conf --pid-file=/var/lib/nova/networks/nova-br1002.pid --listen-address=10.10.1.3 --except-interface=lo --dhcp-range=set:franknet,10.10.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br1002.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br1002.hosts\nExit code: 2\nStdout: u\'\'\nStderr: u"2015-01-27 13:29:13.983 10307 INFO nova.openstack.common.periodic_task [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Skipping periodic task _periodic_update_dns because its interval is negative\\n2015-01-27 13:29:13.985 10307 INFO nova.network.driver [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Loading network driver \'nova.network.linux_net\'\\n2
Once I killed the offending dnsmasq process, I could successfully create instances.
SOURCE: http://fedoraproject.org/wiki/QA:Testcase_launch_an_instance_on_OpenStack
1: Deployment Server (where I could then run the 'ds wizard' command)
2: My all-in-one deployment to a KVM hypervisor.
Both machines are running RHEL 6.5 and have lots of CPU, RAM and disk.
My HUGE problem after install was that the Deployment Server automatically started an unnecessary dnsmasq process. And because of this, I could not create any new instances. Every time I tried, I got errors similar to the following in /var/log/nova/compute.log (and similar in network.log also):
2015-01-27 13:29:14.987 19521 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Remote error: ProcessExecutionError Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=3 dnsmasq --strict-order --bind-interfaces --conf-file=/etc/dnsmasq.conf --pid-file=/var/lib/nova/networks/nova-br1002.pid --listen-address=10.10.1.3 --except-interface=lo --dhcp-range=set:franknet,10.10.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br1002.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br1002.hosts
Exit code: 2
Stdout: u''
Stderr: u"2015-01-27 13:29:13.983 10307 INFO nova.openstack.common.periodic_task [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Skipping periodic task _periodic_update_dns because its interval is negative\n2015-01-27 13:29:13.985 10307 INFO nova.network.driver [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Loading network driver 'nova.network.linux_net'\n2015-01-27 13:29:13.986 10307 DEBUG nova.servicegroup.api [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.6/site-packages/nova/servicegroup/api.py:65\n2015-01-27 13:29:14.107 10307 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.6/site-packages/stevedore/extension.py:156\n2015-01-27 13:29:14.121 10307 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/lib/python2.6/site-packages/stevedore/extension.py:156\n\ndnsmasq: failed to create listening socket for 10.10.1.3: Address already in use\n"
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 193, in deallocate_for_instance\n super(FloatingIP, self).deallocate_for_instance(context, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 563, in deallocate_for_instance\n instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 246, in deallocate_fixed_ip\n address, instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 980, in deallocate_fixed_ip\n self._teardown_network_on_host(context, network)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1906, in _teardown_network_on_host\n self.driver.update_dhcp(elevated, dev, network)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1004, in update_dhcp\n restart_dhcp(context, dev, network_ref)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1118, in restart_dhcp\n _execute(*cmd, run_as_root=True)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1211, in _execute\n return utils.execute(*cmd, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute\n return processutils.execute(*cmd, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 196, in execute\n cmd=sanitized_cmd)\n', u'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=3 dnsmasq --strict-order --bind-interfaces --conf-file=/etc/dnsmasq.conf --pid-file=/var/lib/nova/networks/nova-br1002.pid --listen-address=10.10.1.3 --except-interface=lo --dhcp-range=set:franknet,10.10.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br1002.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br1002.hosts\nExit code: 2\nStdout: u\'\'\nStderr: u"2015-01-27 13:29:13.983 10307 INFO nova.openstack.common.periodic_task [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Skipping periodic task _periodic_update_dns because its interval is negative\\n2015-01-27 13:29:13.985 10307 INFO nova.network.driver [req-0d06a702-1f2a-4b95-a8f3-0f8665d4c83b None None] Loading network driver \'nova.network.linux_net\'\\n2
Once I killed the offending dnsmasq process, I could successfully create instances.
SOURCE: http://fedoraproject.org/wiki/QA:Testcase_launch_an_instance_on_OpenStack
Saturday, January 24, 2015
How to remove a bridged network interface in Red Hat Linux
So ICO 2.4 requires the Deployment Server machine to have interface br0 defined as a bridge connected to eth0. However, ICO 2.4.0.1 does NOT want br0, but wants eth0. I had already created br0 using the virsh command, so I had to look up how to remove it, and it's very simple once you find it:
virsh iface-unbridge br0
and that's it. It puts all of the configuration back on eth0 and you're in business.
virsh iface-unbridge br0
and that's it. It puts all of the configuration back on eth0 and you're in business.
Thursday, January 22, 2015
Getting IBM Cloud Orchestrator to install on Red Hat Enterprise Linux 6.6
This is probably fixed in ICO 2.4.1, but in 2.4, you need to modify a file to get the installation to work on RHEL 6.6 because it believes that only RHEL 6.4 and 6.5 are supported. You'll encounter an error when you run through the ds wizard command.
The file you need to update (after installing the Deployment Service) is:
/opt/ibm/cloud-deployer/precheck.json
In that file, you'll see multiple stanzas similar to this:
"os": {
"release": "Red Hat Enterprise Linux Server release",
"arch": "64",
"version": ["6.5","6.4"]
The file you need to update (after installing the Deployment Service) is:
/opt/ibm/cloud-deployer/precheck.json
In that file, you'll see multiple stanzas similar to this:
"os": {
"release": "Red Hat Enterprise Linux Server release",
"arch": "64",
"version": ["6.5","6.4"]
},
You need to update the line that contains "version" to be:
"version": ["6.6","6.5","6.4"]
And then the wizard can complete.
Some background:
I found this because I saw the error:
"DeployTaskFailed: Failed to execute deployment task: deploy-precheck, error message: precheck failed on ico24demo.mynet.foo: ico24demo.mynet.fooincorrect os version,expected:['6.5', '6.4']actually:6.6\n\n"
And I ran the following command in multiple directories:
grep -r "6\.5" *
until I found the file I needed to change.
Configuring RHEL prerequisites for IBM Cloud Orchestrator 2.4
Introduction
The first basic requirement for your systems when deploying ICO 2.4 is to have DNS name resolution working correctly. For some reason or another, this is the area that has the largest number and most frequent problems in every single enterprise. So this short post is meant to help you get something going quickly, normally in a test environment, until you can communicate with your networking team to get the problems solved permanently.My Simple Environment
My test environment is the demo configuration with one RHEL 6.6 server with 8 cores and 32GB of RAM, with KVM for virtualization.Setting Your Hostname
You need to set your hostname to a Fully Qualified Domain Name (FQDN), and in a test environment, the format just needs to be correct - the actual name doesn't have to be registered anywhere. For example, I named my host ico24demo.mynet.foo. I don't care if .foo is a valid root or not because I get to make the rules in my own environment.To permanently set your hostname to an FQDN if you didn't do it at install time, you need to edit the file:
/etc/sysconfig/network
and set:
HOSTNAME=your.full.fqdn
If you don't want to reboot to have it set, also run the following as root:
hostname your.full.fqdn
Update Your Hosts File
You also need to update your /etc/hosts file with your hostname and IP. So edit /etc/hosts and add that information for your host. In my environment, my hostname is ico24demo.mynet.foo and my static IP address is 192.168.1.250. So here's my /etc/hosts entry:192.168.1.250 ico24demo.mynet.foo ico24demo
Start dnsmasq
dnsmasq is a simple DNS server (and DHCP and some other things) that we'll configure locally for name resolution.
rpm -q -a | grep dnsmasq
If it's not installed, install it.
Assuming it is installed, make sure it's not already running with:
ps -ef | grep dnsmasq
If it is, kill it with:
service stop dnsmasq
The above may or may not work. If the process is still running kill it with the kill command.
And now that your /etc/hosts file has your host information, start dnsmasq with:
service start dnsmasq
Test Your New DNS Server
Probably the easiest way to test your server is to first edit your /etc/resolv.conf file to set your nameserver to your local machine. So my local IP is 192.168.1.250, and the IP address of my subnet's nameserver is 192.168.1.1. So in my /etc/resolv.conf file, I have the following:# Generated by NetworkManager
nameserver 192.168.1.250
nameserver 192.168.1.1
host 192.168.1.250
The correct output for me is:
250.1.168.192.in-addr.arpa domain name pointer ico24demo.mynet.foo.
If you get something different, go back over the steps above. But if it's correct, keep going.
Make the Nameserver Change Permanent
Go to System->Preferences->Network Connections to set your nameserver to be your local IP address.A non-GUI way to do this is to update your /etc/sysconfig/network-scripts/ifcfg-eth0 file to set the DNS server.
Good luck!
Subscribe to:
Posts (Atom)