CloudStack 2.14 + KVM + Openwitch + openflow (Part 3)

Part 3: Install and configure CloudStack Agent with NFS storage and Openvswitch bridge.

In the part 1 and part 2, I have introduced the basic install of CentOS, Openvswitch, CloudStack manager. In this part, I will introduce the CloudStack Agent installation with OVS bridge and NFS storage

After the end of part 2, Host1 and Host2 configuration as following:

Host1

• Manager interface: cloudbr0 (eth0) - IP: 10.0.0.2/24
• Guest and Public interface: cloudbr1 (eth1)
• Sorfware installed: CentOS 6.0, OVS, CloudStack manager and database

Host2

• Manager interface: cloudbr0 (eth0) - IP: 10.0.0.3/24
• Guest and Public interface: cloudbr1 (eth1)
• Sorfware installed: CentOS 6.0, OVS

You have built a NFS server with IP: 10.0.0.4/24 with URL: 10.0.0.4/kvm-ovs. This link will use for Primary storage on KVM cluster.

Now, You must install cloudstack on both hosts.

• Check for a fully qualified hostname.

    #hostname --fqdn

• This should return a fully qualified hostname such as "hostX.cloudkvm.com". If it does not edit /etc/hosts so that it does.
• Remove qemu-kvm. The CloudStack provides a patched version.

    #yum erase qemu-kvm

•     Disable SELinux

    Set SELinux up to be permissive by default. First, set the SELINUX variable in /etc/selinux/config to “permissive”. This ensures that the CloudStack Agent can run properly on system reboot. Then set SELinux to permissive until the system is rebooted:

    # setenforce permissive

    Install the CloudStack packages. You should have a file in the form of CloudStack-NNNN.tar.gz. Untar the file and and then run the install.sh script inside it.

    # tar xzf CloudStack-2.2.0-1-centos.tar.gz

    # cd CloudStack-2.2.0-1-centos

    # ./install.sh

    Setting up the temporary repository...

    Cleaning Yum cache...

    Loaded plugins: fastestmirror

    11 metadata files removed

    Welcome to the Cloud.com CloudStack Installer.  What would you like to do?

        M) Install the Management Server

        A) Install the Agent

        S) Install the Usage Monitor

        D) Install the database server

        Q) Quit

        > A

    Choose A to install the Agent software 

The CloudStack Agent is now installed. Later in the installation you will add this host to the CloudStack via the Management Server. This step will configure the Agent on the Host. 

The next, you must edit configuration file of CloudStack Agent as following:

#Fri Nov 02 16:38:04 ICT 2012

guest.network.device=cloudbr1

workers=5

private.network.device=cloudbr0

port=8250

resource=com.cloud.agent.resource.computing.LibvirtComputingResource

pod=1

zone=1

guid=c3434f39-0252-3c45-9a42-a713d5bcf456

public.network.device=cloudbr1

cluster=1

local.storage.uuid=80c6bd25-449c-492b-b2c1-43b6abd5ca81

LibvirtComputingResource.id=1

host=10.0.0.X

Note:

To CloudStack Agent can run with OVS bridge you must check brctl command:

#brctl addbr testbridge  => successful with no error

#brctl delbr testbridge   => delete testbridge

Now, you must download CloudStack-NNNN from source code and change some files and rebuild it from source.

/CloudStack-NNN/scripts/vm/network/vnet/modifyvlan.sh, 

/CloudStack-NNN/python/lib/cloud_utils.py

/CloudStack-NNN/target/scripts/util/qemu-ifup

/CloudStack-NNN/core/src/com/cloud/agent/resource/virtualnetwork/VirtualRoutingResource.java

=> You find the command brctl and replace by the ovs-vsclt command with the command structure of the ovs-vsclt

Example: "brctl show |grep bridge_name"  to "ovs-vsctl show |grep  bridge_name"

Now, you can add Host1 and Host2 and create VMs.

Part 4: Create VM with OVS bridge and configure QoS and network throttling on VMs

CloudStack 2.14 + KVM + Openwitch + openflow (Part 2)

Part 2: Create bridge interface (use openvswitch bridge) for Cloud agent and configure agent.

Step 1: Create bridge interface for CloudStack.

Reference links: http://cloudstack.org/, http://openvswitch.org/

In part 1, I have introduced the basic installs for Host 1 and Host 2, in the second part I will introduce the network configuration on hosts to prepare for install Cloudstack and KVM.

Host 1 and Host 2 have 2 interface eth0 and eth1.

You must configure eth0 for manager interface (cloudbr0) => 10.0.0.2 for Host1 and 10.0.0.3 for Host2

                              eth1 for guest and public interface (cloudbr1)

You must perform the following commands on both Hosts:

Create parent bridge on openvswitch:

ovs-vsctl add-br br0

cloudbr0:

ovs-vsctl add-br cloudbr0 br0 => Create cloudbr0 bridge

ovs-vsctl add-port cloudbr0 eth0 => Add eth0 interface to cloudbr0 bridge

ifconfig cloudbr0 10.0.0.2/24 up => on Host1

ifconfig cloudbr0 10.0.0.3/24 up => on Host2

cloudbr1

ovs-vsctl add-br cloudbr1 br0 => Create cloudbr1 bridge

ovs-vsctl add-port cloudbr1 eth1 => Add eth1 interface to cloudbr1 bridge

You can check the bridge of openvswitch which ran with CloudStack by adding a bridge interface (br-ovs example):

brctl addbr br-ovs  => Ok with no error. If you see any error, please check compat module on OVS.

You have finished setup bridge interface on OVS for CloudStack Agent.

Step 2: Install CloudStack manager

Install Manager

Install the CloudStack packages. You should have a file in the form of CloudStack-NNNN.tar.gz. Untar the file and the run the install.

Choose "M" to install the Management server software.

Install Database

Run install.sh again and choose "D" to install MySQL

# ./install.sh

Setting up the temporary repository...

Cleaning Yum cache...

Loaded plugins: fastestmirror

11 metadata files removed

Welcome to the Cloud.com CloudStack Installer.  What would you like to do?

    A) Install the Agent

    S) Install the Usage Monitor

    D) Install the database server

    U) Upgrade the CloudStack packages installed on this computer

    R) Stop any running CloudStack services and remove the CloudStack packages from this computer

    Q) Quit

    > D

Edit the MySQL configuration (/etc/my.cnf) and insert the following lines in the [mysqld] section. You can put these lines below the datadir line.

innodb_rollback_on_timeout=1

innodb_lock_wait_timeout=600

max_connections=350

Restart the MySQL service and configure it to start on boot

# service mysqld restart

# chkconfig mysqld on

Use the following script to create the cloud user on the database with a password of your choice. This script deploys the database using the credentials in the deploy-as parameters. By default on RHEL/CentOS/Fedora MySQL is installed with no root password. If you have set a password you will need to postpend that to the deploy-as parameter as in "--deploy-as=root:password

# cloud-setup-databases cloud:dbpassword@localhost --deploy-as=root
Final, you must run #cloud-setup-management

Part 3: Install and configure Agent with NFS storage and openvswitch bridge.

CloudStack 2.14 + KVM + Openwitch + openflow (Part 1)

Part 1: Install centos 6 and openvswitch

I have built a private cloud system with Cloudstack 2.14 and KVM on centos 6.0. However, the CS2.14 not support the bandwidth limit (network throttling) on the VMs.The bandwidth limit is only supported on the CS 4.1 http://bugs.cloudstack.org/browse/CS-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel.I have chosen the solution is install Openvswitch on KVM . Openvswitch + openflow to create a virtual network, QoS, firewall ... for VMs

I used the following link in my lab:
- Install openvswitch and KVM on centos:  http://networkstatic.net/installing-openvswitch-and-kvm-on-centos-6-2-part-1/
- Install openvswitch http://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL;hb=HEAD
- Cloudstack document  http://docs.cloudstack.org/

My lab:
 - 1 PC with iperf installed.
 - Host 1 IBM X3650 (Install Centos 6.0, CloudStack 2.14 Manager, KVM Agent, openvswitch with openflow)
 - Host 2 IBM X3650 (Install Centos 6.0, KVM Agent, openvswitch with openflow)


1. Install base package on Host 1 and Host 2
Step 1: Install Centos 6.0 at link: http://www.itbox4vn.com/2011/08/how-to-install-centos-6-step-by-step.html

Step 2: Install Openvswitch
Download the following tars and packages to resolve dependencies or packages not in the Yum repositories.

DL Openvswitch tar

$ wget http://openvswitch.org/releases/openvswitch-1.3.0.tar.gz

$ wget http://www.graphviz.org/pub/graphviz/stable/SRPMS/graphviz-2.28.0-1.src.rpm

$ wget http://www.graphviz.org/pub/graphviz/stable/redhat/el6/i386/os/webdot-2.26-1.el6.noarch.rpm

wget http://download.fedoraproject.org/pub/fedora/linux/releases/16/Everything/source/SRPMS/autoconf-2.68-2.fc15.src.rpm

$rpm -ivh graphviz-2.28.0-1.src.rpm

$rpm -ivh webdot-2.26-1.el6.noarch.rpm

$yum -y install graphviz-devel

$rpm -ivh webdot-2.26-1.el6.noarch.rpm

$yum -y install graphviz-tcl

$yum -y install python-zope-filesystem

$yum -y install python-zope-interface

$yum -y install automake

$yum -y install autoconf

add yum -y install openssl-devel this package is necessary if you use SSL with OpenVSWitch. (Thanks for input Marcos)

$yum –y install gcc
The Open vSwitch datapath requires bridging support
(CONFIG_BRIDGE) to be built as a kernel module.  (This is common
in kernels provided by Linux distributions.)  The bridge module
must not be loaded or in use.  If the bridge module is running
(check with “lsmod | grep bridge”), you must remove it (“rmmod
bridge”) before starting the datapath.

To build the Linux kernel module, so that you can run the
kernel-based switch, pass the location of the kernel build
directory on –with-linux.  For example, to build for a running
instance of Linux:

./configure –with-linux=/lib/modules/`uname -r`/build

$make

$make install

Load the built kernel module

insmod datapath/linux/openvswitch_mod.ko
insmod datapath/linux/brcompat_mod.ko

Verify the kernel mod loaded
[root@localhost openvswitch-1.3.0]# lsmod
openvswitch_mod        82086  0

If everything compiles and install create the sockets and db.

$mkdir -p /usr/local/etc/openvswitch

$ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema

Start the configuration DB
$
ovsdb-server /usr/local/etc/openvswitch/conf.db
–remote=punix:/usr/local/var/run/openvswitch/db.sock
–remote=db:Open_vSwitch,manager_options
–private-key=db:SSL,private_key
–certificate=db:SSL,certificate
–bootstrap-ca-cert=db:SSL,ca_cert –pidfile –detach –log-file

Initialize the DB
$ovs-vsctl –no-wait init

Then start the main Open vSwitch daemon, telling it to connect to the
same Unix domain socket
$ovs-vswitchd –pidfile –detach
$sudo ovs-vswitchd –pidfile –log-file –detach -v
$ovs-vswitchd –pidfile –detach

Part 2: Create bridge interface (use openvswitch bridge) for Cloud agent