How to deploy a xifi node

From wiki
Jump to: navigation, search

This HOW-to aims to describe how to go from bare metal to a fully federated infrastructure.


If you encounter problems, you are to submit a TICKET via If you use ITbox, the "Generate Diagnostic Snapshot" will collect all / most logs that the developers need.


The following assumptions are made;

  1. FUEL/ITbox
  2. You have hardware to deploy on. Minimum, four computers (nodes); one to host FUEL/ITbox, one to act as Controller, one to act as Monitor, and one or more to act as Controller.
  3. You have a working MDPVN (or alternative to reach the MDPVN) and public IP addresses to assign to your OpenStack Nodes (not crucial during deployment, but for the federation to work).
  4. You will deploy Neutron with GRE segmentation.
  5. Layer 2 config.
    • If you have a simple layer 2 network, ie. one LAN Segment that connects the FUEL/ITBox to the nodes, this will be used to PXE boot the nodes during Deployment. The other segment needs to support VLAN tagging, default vlans are 100 and 101. The default VLAN should be untagged.
    • In case your nodes only have one nic, then a third vlan is needed for the public network.
  6. Information collection; You need the following information
    • RegionID; AFAIK, this is the 'city' where the infrastructure is located in (how to handle multiple infrastructures in the same region?)
    • .... (please complete)

Installation of FUEL/ITbox[edit]

  1. Download FUEL/ITBox from;, burn to CD/USB boot and install on the allocated hardware.
    • If your FUEL/ITbox node has multiple nics, do not config using DHCP, configure static address(es).
    • If you will deploy on Dell devices with PERC or Matrox/nVidia cards, patch the GRUB kernel thats deployed to the nodes (
    • Make sure that no DHCP servers are on the LAN where your nodes will communicate (at least during deployment)
  1. When FUEL/ITbox is installed, during the first boot make sure that you configure FUEL, then reboot.
    • Possibly patch the puppetdb memory; 'sed -i.bak s/Xmx192m/Xmx2g/g /etc/sysconfig/puppetdb '
  2. Configure the other nodes to always PXE boot, you man possibly disable memory check.
  3. Once the FUEL/ITbox is operational (web interface accessible), boot the other nodes.

OpenStack Deployment[edit]

  1. Create your OpenStack Environment, from the FUEL/ITbox interface.
    1. Name and Release: Pick a Nice name, select Grizzly on Ubuntu 12.04 (2013.1.4).
    2. Deployment: Multi-Node (Keep it simple, for the first time)
    3. Compute: KVM
    4. DCRM GE: IBM DCRM Pivot Scheduler
    5. Network: Neutron with GRE segmentation
    6. Finish: Create
  1. Configure your environment
    1. In the Nodes tab; add your nodes and associate the correct computers with them.
    2. Check that the networks are correctly assigned to the correct interfaces. In the simple case, eth0 would hold Admin (PXE) and eth1 Public, Storage (vlan 100) and Management (vlan 101). In the single NIC case; eth0 would hold Admin (PXE), Public (vlan X), Storage (vlan 100), Management (vlan 101).
    3. In the Network Tab;
      • Public Network; use either your white IPs or some IPs from your MDVPN range. I've used the parts of our MDVPN range ( (during deployment). Make sure that your nodes can access the internet via the gateway that you provide for the Public Network. In my case the netmask is, gw
    4. Unless you need, keep the default values for the Management and Storage networks, same goes for Neutron L2 Config.
    5. Neutron L3 Config
      1. External network, these addresses should be on the same network as in the Public range, but used in the public range. In our case,
      2. Internal network, unless you need to, keep the default values. The same applies for name servers.
    6. In the Settings tab.
      1. Monitoring; check
        1. Federation Monitoring Nodes URL(?)
        2. OpenStackDataCollector,
        3. NGSI Adaptor,
        4. Contect Broker,
        5. Nagios
          • If your Compute Node failes, with something like "err: Duplicate declaration: Notify[undef] is already declared in file /etc/puppet/modules/nagios/manifests/common.pp at line 84; cannot redeclare at /etc/puppet/moduless/nagios/manifests/common.pp:84 on node .... Follow this patch.
      2. Federation; fill in your information.
      3. Access; possibly an appropriate email address.
      4. Common; Select
        1. Hypervisor type- KVM
        2. Auto Assign float IP
        3. Enable 'OVS VLAN Spliters'
        4. Use qcow format for images
        5. Start guests on boot
        6. Scheduler driver; Pivot scheduler
      5. Syslog and Storage keep the defaults (unless you need to change)
  2. Deploy your environment, wait...Have Nice day

Verify the deployment[edit]

  1. Logon to your FUEL/ITbox,
  2. Issue "fuel node list"; the status of the nodes should be "Ready".
[root@fuel ~]# fuel node list
id | status | name             | cluster | mac               | roles                          | pending_roles | online
49 | ready  | Untitled (49:98) | 12      | D4:AE:52:D4:49:98 | [u'cinder']                    | []            | True  
50 | ready  | Untitled (C9:30) | 12      | D4:AE:52:EA:C9:30 | [u'compute']                   | []            | True  
51 | ready  | Untitled (C4:9E) | 12      | D4:AE:52:EA:C4:9E | [u'compute']                   | []            | True  
52 | ready  | Untitled (0C:0D) | 12      | C4:54:44:33:0C:0D | [u'monitoring', u'controller'] | []            | True  
[root@fuel ~]#
  1. Logon using ssh to the controller node; ssh node-<id of controller node, 52 in example>
  2. source the openrc ". openrc", execute "nova service-list"
root@node-52:~# nova service-list
| Binary           | Host    | Zone     | Status  | State | Updated_at                 |
| nova-cert        | node-52 | internal | enabled | up    | 2014-09-11T20:35:43.000000 |
| nova-compute     | node-50 | nova     | enabled | up    | 2014-09-11T20:35:40.000000 |
| nova-compute     | node-51 | nova     | enabled | up    | 2014-09-11T20:35:42.000000 |
| nova-conductor   | node-52 | internal | enabled | up    | 2014-09-11T20:35:45.000000 |
| nova-consoleauth | node-52 | internal | enabled | up    | 2014-09-11T20:35:43.000000 |
| nova-scheduler   | node-52 | internal | enabled | up    | 2014-09-11T20:35:42.000000 |
  1. You should see your compute nodes (50 and 51 in example).
  2. Use cinder-manage host list, to see your cinder hosts (?)
root@node-52:~# cinder-manage host list 
host                            zone           
node-52                         nova           
node-49                         nova           

Configure Secondary Network[edit]

This depends _massively_ on your physical/logical layout. In this example, this is as follows:


Node -- em0 -- PXE -->ITBox
     -- em1.104 --> PUBLIC MDVPN (as defined in ITbox) -->+----+--> mdvpn
        em1.100 --> Management                            |    | 
        em1.101 --> Storage                               |GW  |    
        em1 (untagged) --> PUBLIC ----------------------->+----+---> white

The ITbox install creates br-ex on all nodes; You need to do the same for another network. As you might need to do this for many nodes, here is a script that you'd execute on that node.


echo "Setting up secondary External Network";

mdvpnIP=$(ifconfig br-ex | grep 'inet addr:' | awk '{print $2}' | sed 's/addr://' )
hostID=$(ifconfig br-ex | grep 'inet addr:' | awk '{print $2}' | sed 's/addr://'  | awk -F '.' '{print $4}')


echo "Host is known as $mdvpn, with id $hostID"

echo "Setting up bridge";

ovs-vsctl add-br br-ex-2
ovs-vsctl add-port br-ex-2 br-ex-2--br-eth1
ovs-vsctl set interface br-ex-2--br-eth1 type=patch
ovs-vsctl set interface br-ex-2--br-eth1 options:peer=br-eth1--br-ex-2
ovs-vsctl add-port br-eth1 br-eth1--br-ex-2
ovs-vsctl set interface br-eth1--br-ex-2 type=patch
ovs-vsctl set interface br-eth1--br-ex-2 options:peer=br-ex-2--br-eth1
echo "auto br-ex-2" > /etc/network/interfaces.d/ifcfg-br-ex-2

echo "Allocating IP; $publicNet.$hostID to new bridge."
echo "iface br-ex-2 inet static" >> /etc/network/interfaces.d/ifcfg-br-ex-2
echo "address $publicNet.$hostID" >> /etc/network/interfaces.d/ifcfg-br-ex-2
echo "netmask $publicNetMask" >> /etc/network/interfaces.d/ifcfg-br-ex-2
echo "gateway $publicNet.$routerID" >> /etc/network/interfaces.d/ifcfg-br-ex-2

echo "Removing old default GW"

sed -i "s/gateway/#gateway/g" /etc/network/interfaces.d/ifcfg-br-ex
echo "up route add -net netmask gw $mdvpnGW" >> /etc/network/interfaces.d/ifcfg-br-ex

echo "Patch  /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini"
echo "create another phynet3, OpenStack will connect its network to this. "
sed -i.bak "s/physnet1,physnet2/physnet1,physnet2,physnet3/g" /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
sed -i.bak "s/physnet2:br-prv/physnet2:br-prv,physnet3:br-ex-2/g" /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

echo "Restart ovswitch plugin"
service quantum-plugin-openvswitch-agent restart

One the node has updated its network settings, the basics are done. I.e your nodes will be connected to br-ex-2, and should reach the both GWs. There should be a route for the MDVPN, and a default.

On the controller, you create a network that is a flat, physical network with an external router. Create a subnet associated with it as well. You can also create a router that attach to the network as well. I did it via the horizon gui.

quantum net-create --provider:network_type=flat --provider:physical_network=physnet3 --router:external=true public_ext
quantum subnet-create --allocation-pool start=,end= --gateway public_ext

At this point, you can follow Xifi:Wp5: OpenStack Customizations from L3 Agent.


grep type_drivers /etc/neutron/plugin.ini ##type_drivers ==> gre,flat,local
grep flat_networks /etc/neutron/plugin.ini ##flat_networks ==> *

Identify the interface that holds the public network, may be eth0,eth1,eth2 or eth3. Adapt the stuff below accordingly.

tail /etc/neutron/plugin.ini ## should not incl. the stuff below.
echo "bridge_mappings = phy-ex:br-ex,phy-ex-2:br-ex-2" >>  /etc/neutron/plugin.ini
echo "network_vlan_ranges = phy-ex:114:114" >>  /etc/neutron/plugin.ini
ovs-vsctl add-br br-ex-2
ovs-vsctl add-port br-ex-2 br-ex-2--br-eth1
ovs-vsctl set interface br-ex-2--br-eth1 type=patch 
ovs-vsctl set interface br-ex-2--br-eth1 options:peer=br-eth1--br-ex-2
ovs-vsctl add-port br-eth1 br-eth1--br-ex-2
ovs-vsctl set interface br-eth1--br-ex-2 type=patch
ovs-vsctl set interface br-eth1--br-ex-2 options:peer=br-ex-2--br-eth1

    1. See for IP


echo "auto br-ex-2 " > /etc/network/interfaces.d/ifcfg-br-ex-2
echo "iface br-ex-2 inet static " >> /etc/network/interfaces.d/ifcfg-br-ex-2
echo "address " >> /etc/network/interfaces.d/ifcfg-br-ex-2
echo "netmask " >> /etc/network/interfaces.d/ifcfg-br-ex-2
echo "gateway " >> /etc/network/interfaces.d/ifcfg-br-ex-2

in /etc/network/interfaces.d/ifcfg-br-ex-2

echo "up route add -net netmask gw ">> /etc/network/interfaces.d/ifcfg-br-ex

NFS and Nagios on NODES[edit]

##INSTALL NFS to simplify life..
apt-get install nfs-common && echo "       /mnt/node_root  nfs     defaults        0       0" >> /etc/fstab && mkdir /mnt/node_root && mount /mnt/node_root

##IPTABLES may need update
##Add the std. sources
echo " " >> /etc/apt/sources.list
cat /mnt/node_root/add.sources.list >> /etc/apt/sources.list
apt-get update
apt-get install emacs libssl-dev nagios-plugins nagios-nrpe-plugin nagios-nrpe-server nagios-plugins-contrib
##in /etc/nagios/nrpe.cfg, change 
##allowed_hosts= to >> allowed_hosts=,,
sed -i "s/allowed_hosts=,," /etc/nagios/nrpe.cfg
##Add the NRPE stuff for all, then the CONTROLLER needs more
cat /mnt/node_root/add.nrpe_local.cfg_compute_nodes >> /etc/nagios/nrpe_local.cfg
cp /mnt/node_root/add.check_hostname /usr/lib/nagios/plugins/check_hostname
chmod a+rwx /usr/lib/nagios/plugins/check_hostname
##VERIFY that disc is correct == check_hda1

Volume management (cinder)[edit]

The nova-api and nova-compute should acquire the cinder endpoint from the service catalog. This does not happen currently as explained in Xifi:Wp5:_OpenStack_Customizations#Issue_observed_-_cinder_endpoint_missing_due_to_failing_to_fetch_the_service_catalog. To work around this issues one needs to override the service catalog lookup by specifying the endpoint url and providing an OS region name (which prevents a runtime exception).

On the controller node as well as on the compute nodes we have modified the DEFAULT section in /etc/nova/nova.conf to contain the following two lines:


where is the private IP address of the controller node, where the endpoint resides.

All nova and cinder services must be restarted on each node where the configuration was changed.

NOTE 1: Depending on your setup you may need to a public IP address instead of a private one on the controller node. The important part here is that the compute nodes and controller are able to reach each other as well as the IP address for the cinder endpoint.

NOTE 2: Using (project_id) instead of (tenant_id) in the cinder_endpoint URL is required to make volume management working

NOTE 3: We combine cinder and compute roles on the same node. It is unclear if this approach works when you allocate the roles differently.

NOTE 4: When you attach the volume to a running VM instance it gets attached to the first /dev/vdX device, ignoring what was requested in the Cloud Portal. See Xifi:Wp5:_OpenStack_Customizations#Issue_observed_-_volume_attaching_to_the_wrong_device for details.

Installing and configuring additional components[edit]

This is mainly extracted from Deliverable D2.2, section

  1. Install ContextBroker; unless the install worked via ITBox. See
    • Verify that its installed and auto-starting upon reboot (chkconfig or update-rd.d)
    • Register/Federate ContextBroker
  2. Install BigData GE (COSMOS), see
  3. Install nsgi2cosmos connector, see


Misc Config files[edit]

Some works, some are here for debugging.

BTH_quantum_controller /etc/quantum/* (for debugging, secondary network)