Archives For ubuntu

kvm-logo-squareAs KVM seems more and more interesting, at work we wanted to do a Proof-of-Concept. The KVM hypervisor cluster had to be controlled by CloudStack and also integrate with NSX (formerly known as Nicira).

NSX is owned by VMware these days and is one of the first Software Defined Networking solutions. At Schuberg Philis we use this since early 2012.

Choosing an OS
To me, the most interesting part of KVM is the fact you only need a very basic Linux box with some tooling and you have a nice modern hypervisor ready to rock. Since we’re using CloudStack to orchestrate everything, we do not need cluster features. In fact, this prevents the “two captain” problem that we sometimes encounter with XenServer and VMware ESX. We compared Ubuntu with CentOS/RHEL and both work fine. It all depends on your needs.

Installing KVM
Installing the software is pretty straight forward:


yum install kvm libvirt python-virtinst qemu-kvm bridge-utils pciutils


apt-get install qemu-kvm libvirt-bin bridge-utils virt-manager openntpd

Installing Open vSwitch
Open vSwitch is a multilayer virtual switch and it brings a lot of flexibility in the way you can create interfaces and bridges in Linux. There are two options here. If you need STT tunnels, you need the NSX patched version of Open vSwitch. If you need VXLAN or GRE tunnels, you can use the open source version that comes with Ubuntu and CentOS. Both ship version 2.3.1 which works perfectly fine.


yum install openvswitch kmod-openvswitch


apt-get install openvswitch-switch

Configuring Open vSwitch
Instead of classic Linux bridges, we use Open vSwitch bridges. In our POC lab environment, we were using HP DL380 G9 servers that have 2 10Gbit NICs to two Arista switches. They run a LACP bond and on top of this we create the bridges for KVM to use. Because we setup the Open vSwitch networking over and over again while debugging and testing different OS’es, I wrote a script that can quickly configure networking. You can find it at Github.

To give some quick pointers:

Create a bridge:

ovs-vsctl add-br cloudbr0

Create an LACP bond:

ovs-vsctl add-bond cloudbr0 bond0 eno49 eno50\
bond_mode=balance-tcp lacp=active other_config:lacp-time=fast

Create a so-called fake bridge (with a VLAN tag):

ovs-vsctl add-br mgmt0 cloudbr0 123

Get an overview of current configuration:

ovs-vsctl show

Get an overview of current bond status:

ovs-appctl show/bond

Example output:

---- bond0 ----
bond_mode: balance-tcp
bond may use recirculation: yes, Recirc-ID : 300
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 9887 ms
lacp_status: negotiated
active slave mac: fc:00:00:f2:00(eno50)

slave eno49: enabled
 may_enable: true
 hash 139: 3 kB load

slave eno50: enabled
 active slave
 may_enable: true
 hash 101: 1 kB load
 hash 143: 5 kB load
 hash 214: 5 kB load
 hash 240: 5 kB load

Add hypervisor to NSX
For now I assume you already have a NSX cluster running capable of acting as a controller/manager for Open vSwitch. If you don’t know NSX, have a look because it’s awesome.

We do need to connect our Open vSwitch to the NSX cluster. To do that, you need a SSL certificate. This is how you generate one:

cd /etc/openvswitch
 ovs-pki req ovsclient
 ovs-pki self-sign ovsclient
 ovs-vsctl -- --bootstrap set-ssl \
 "/etc/openvswitch/ovsclient-privkey.pem" \
 "/etc/openvswitch/ovsclient-cert.pem" \

Next, add the hypervisor to NSX. You can either set the authentication to be ip-address based (it will then exchange certificates on connect) or copy/paste the certificate (ovsclient-cert.pem) to NSX at once. The first method allows for easier automation. I’m showing the UI here, but of course you can also use the API to add the hypervisor.

Setting authentication to be ip-address based (allow for automatic exchange of security certificates)

Setting authentication to be ip-address based (allow for automatic exchange of security certificates)


Setting authentication to use a security certificate (and provide one manually)

Setting authentication to use a security certificate (and provide one manually)


The final step is to connect Open vSwitch to NSX:

ovs-vsctl set-manager ssl:

Then NSX should show green lights and tunnels are being created.

To get an idea of what’s going on, you can run:

ovs-vsctl list manager
ovs-vsctl list controller
ovs-vsctl show

Debugging this can be done from the command line (check Open vSwitch logs) or from NSX.

Debugging in NSX is very nice

Debugging in NSX is very nice


At this point, NSX is controlling our Open vSwitch.

Setting up the CloudStack agent
When running KVM, CloudStack runs an agent on the hypervisor in order to configure VMs.

Installing the agent is simply installing a RPM or DEB package. Depending on the version you will use different repositories. At Schuberg Philis, we build our own packages that we serve in a repository.

Because we’re using Open vSwitch, some settings need to be tweaked in the file, found in /etc/cloudstack/agent.

echo "" \
>> /etc/cloudstack/agent/
echo "network.bridge.type=openvswitch" \
>> /etc/cloudstack/agent/

You may also want to set the log level to debug:

sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml

CloudStack requires some KVM related settings to be tweaked:

# Libvirtd
echo 'listen_tls = 0' >> /etc/libvirt/libvirtd.conf
echo 'listen_tcp = 1' >> /etc/libvirt/libvirtd.conf
echo 'tcp_port = "16509"' >> /etc/libvirt/libvirtd.conf
echo 'mdns_adv = 0' >> /etc/libvirt/libvirtd.conf
echo 'auth_tcp = "none"' >> /etc/libvirt/libvirtd.conf
# libvirt-bin.conf
sed -i -e 's/libvirtd_opts="-d"/libvirtd_opts="-d -l"/' \
service libvirt-bin restart

# qemu.conf
sed -i -e 's/\#vnc_listen.*$/vnc_listen = ""/g' \

On CentOS 7 Systemd ‘co-mounts’ cpu,cpuacct cgroups, and this causes issues for launching a VM with libvirt. On the mailing list this is the suggested fix

Edit /etc/systemd/system.conf and pass empty string to JoinControllers parameter. Then rebuild the initramfs via ‘new-kernel-pkg –mkinitrd –install `uname -r`’.

Something I currently don’t like: SELinux and AppArmour need to be disabled. I will dive into this and get it fixed. For now, let’s continue:

#AppArmour (Ubuntu)
ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

# SELinux (CentOS)
setenforce permissive
vim /etc/selinux/config

You can now add the host to CloudStack, either via de UI or the API.

Keep an eye on the agent log file:

less /var/log/cloudstack/agent/agent.log

After a few minutes, the hypervisor is added and you should be able to spin up virtual machines! 🙂

When we spin up a VM, CloudStack does the orchestration. So, CloudStack is the one to talk to NSX to provide the network (lswitch) and NSX communicates with Open vSwitch on the hypervisor. The VM is provisioned by CloudStack and KVM/Libvirt makes sure the right virtual interfaces are plugged in Open vSwitch. This way VMs on different hypervisors can communicate over their own private guest network. All dynamically created without manual configuration. No more VLANs!

If it does not work right away, look at the different log files and see what happens. There usually are hints that help you solve the problem.

KVM hypervisors can be connected to NSX and using Open vSwitch you can build a Software Defined Networking setup. CloudStack is the orchestrator that connects the dots for us. I’ve played with this setup for some time now and I find it very fast. We’ll keep testing and probably create some patches for CloudStack. Great to see that the first KVM related pull request I sent is already merged 🙂

Looking forward to more KVM!

Recently I played with Open vSwitch and it’s awesome! Open vSwitch is a multilayer virtual switch and it brings a lot of flexibility in the way you can create interfaces and bridges in Linux. It’s also a Linux distribution independent way to configure these things. Switching in software!

To create a bridge, simply run:

ovs-vsctl add-br remibr0

You can also create another bridge on top of it, to handle a VLAN for example:

ovs-vsctl add-br mgmt0 remibr0 101

Even better, create a bond based on LACP:

ovs-vsctl add-bond remibr0 bond0 em49 em50 bond_mode=balance-tcp lacp=active other_config:lacp-time=fast

This is all quite nice but still basic. It gets interesting when you realise you can connect two switches like you can put a patch cable between physical switches. To test how cross platform this works, I setup two boxes: a CentOS 7 box and a Ubuntu 15.04 one. This shows it in a picture:


We’ll create a new bridge and add a vxlan interface that connects to the other vswitch. Then create a port on it and assign it an ip address. Installing Open vSwitch should be simple, as it is included in the releases.

Create the configuration and be sure to fill in the right ip addresses.

ovs-vsctl add-br remibr0
ovs-vsctl add-port remibr0 vxlan0 -- set Interface vxlan0 type=vxlan options:remote_ip=
ovs-vsctl add-port remibr0 vi0 -- set Interface vi0 type=internal
ifconfig vi0 up

On the second box, bring up on vi0.

Your config should look like this:

ovs-vsctl show
 Bridge "remibr0"
   Port "vxlan0"
     Interface "vxlan0"
       type: vxlan
       options: {remote_ip=""}
   Port "remibr0"
     Interface "remibr0"
       type: internal
   Port "vi0"
     Interface "vi0"
       type: internal
 ovs_version: "2.3.1"

And on the second box:

ovs-vsctl show


 Bridge "remibr0"
   Port "vi0"
     Interface "vi0"
       type: internal
   Port "vxlan0"
     Interface "vxlan0"
       type: vxlan
       options: {remote_ip=""}
   Port "remibr0"
     Interface "remibr0"
       type: internal
 ovs_version: "2.3.90"

As you can see, I used different versions on purpose. You can use two boxes that are the same, of course.

By now, a simple ping test should work:

PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from icmp_seq=2 ttl=64 time=0.009 ms
--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.009/0.014/0.019/0.005 ms

And reversed:

PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=1.47 ms
64 bytes from icmp_seq=2 ttl=64 time=0.202 ms
--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.202/0.839/1.477/0.638 ms

Create a virtual floating ip address
To make the demo a bit more advanced, let’s setup a virtual ip address on the interfaces that can travel between the switches. We use keepalived for this.

vim /etc/keepalived/keepalived.conf

Add this:

global_defs {
 notification_email {
 notification_email_from demo@firewall.loc
 smtp_connect_timeout 30
 router_id LVS_DEVEL
vrrp_instance VI_1 {
 state MASTER
 interface vi0
 virtual_router_id 51
 priority 200
 advert_int 1
 authentication {
 auth_type PASS
 auth_pass 1111
 virtual_ipaddress { 

Copy the config to the other box, be sure to have on MASTER and one BACKUP. Also, the priority of the MASTER should be 200 and the BACKUP 100. It’s just a demo, all it does it bring up an ip address.

Start them both and they should discover each other over the vi0 interfaces on the connected vswitches.

Try pinging the virtual ip address:

PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from icmp_seq=2 ttl=64 time=0.031 ms
64 bytes from icmp_seq=3 ttl=64 time=0.023 ms
--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms

Depending on where the virtual address resides, the latency may be different:

PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.481 ms
64 bytes from icmp_seq=2 ttl=64 time=0.202 ms
64 bytes from icmp_seq=3 ttl=64 time=0.215 ms
64 bytes from icmp_seq=4 ttl=64 time=0.203 ms
--- ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2998ms
rtt min/avg/max/mdev = 0.202/0.275/0.481/0.119 ms

Now start a ping and stop keepalived, then start it again and stop it on the other side. You’ll miss a ping or two when it fails over and then it will recover just fine.

PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.824 ms
64 bytes from icmp_seq=2 ttl=64 time=0.167 ms
64 bytes from icmp_seq=3 ttl=64 time=0.160 ms
64 bytes from icmp_seq=4 ttl=64 time=0.148 ms
64 bytes from icmp_seq=5 ttl=64 time=0.149 ms
From icmp_seq=6 Redirect Host(New nexthop:
From icmp_seq=6 Redirect HostFrom icmp_seq=7 Redirect Host(New nexthop:
From icmp_seq=7 Redirect Host64 bytes from icmp_seq=8 ttl=64 time=0.012 ms
64 bytes from icmp_seq=9 ttl=64 time=0.025 ms
64 bytes from icmp_seq=10 ttl=64 time=0.012 ms
64 bytes from icmp_seq=11 ttl=64 time=0.016 ms
64 bytes from icmp_seq=12 ttl=64 time=0.011 ms
64 bytes from icmp_seq=13 ttl=64 time=0.011 ms
From icmp_seq=14 Redirect Host(New nexthop:
From icmp_seq=14 Redirect HostFrom icmp_seq=15 Redirect Host(New nexthop:
From icmp_seq=15 Redirect Host64 bytes from icmp_seq=16 ttl=64 time=0.323 ms
64 bytes from icmp_seq=17 ttl=64 time=0.162 ms
64 bytes from icmp_seq=18 ttl=64 time=0.145 ms
64 bytes from icmp_seq=19 ttl=64 time=0.179 ms
64 bytes from icmp_seq=20 ttl=64 time=0.147 ms
--- ping statistics ---
20 packets transmitted, 16 received, +4 errors, 20% packet loss, time 19000ms
rtt min/avg/max/mdev = 0.011/0.155/0.824/0.193 ms

Note on the MTU when travelling over the internet
vxlan is encapsulation and this obviously needs space in the packets send over the wire. If you travel over networks that have a default MTU of 1500, it may be wise to lower the MTU of the vi0 interfaces as this will prevent fragmentation. Lowering the MTU is a simple work-around. You could also have a look at GRE tunnels instead.

To alter the MTU:

ip link set dev vi0 mtu 1412

You can make this persistent in Ubuntu’s ‘interfaces’ file and add ‘mtu 1400’. Red Hat alike systems have ‘ifcfg-*’ files for each interface. Add ‘MTU=1400’ to them to alter the MTU.

Although this is a simple demo, the real power comes when you use this to connect two different (virtual or physical) networks in different data centers. You’ll be able to create a Layer-2 network over Layer-3. It’s simple, fast and awesome.

As discussed in a previous post, multi-factor authentication really makes things more secure. Let’s see how we can secure services like SSH and the Gnome desktop with multi-factor authentication.

The Google Authenticator project provides a PAM module than can be integrated with your Linux server or desktop. This PAM module is designed for for home and other small environments. There is no central management of keys, and the configuration is saved in each users home folder. I’ve successfully deployed this on my home server (a Raspberry Pi) and on my work laptop.

When using a Debian-like OS, you can install it with a one-liner

apt-get install libpam-google-authenticator

But note the packaged version is old and does not support all documented options. Below I talk about the ‘nullok’ option, but that is not supported in the packaged version. You then see this error:

sshd(pam_google_authenticator)[884]: Unrecognized option "nullok"
sshd(pam_google_authenticator)[887]: Failed to read "/home/pi/.google_authenticator"

That’s why I suggest building from source, as this can be done quickly:

apt-get remove libpam-google-authenticator
apt-get install libpam0g-dev libqrencode3
cd ~
tar jvxf libpam-google-authenticator-1.0-source.tar.bz2
cd libpam-google-authenticator-1.0/
make install

Open a shell for a user you want to enable two-factor authentication for and run ‘google-authenticator’ to configure.

Configure Google Authenticator

Configure Google Authenticator


Configure your Mobile device
Install the ‘Google Authenticator’ app and just scan the QR-code. It will automatically configure itself and start displaying verification codes.

Verification codes displayed by app

Verification codes displayed by app

You should notice it’ll display new codes each 30 seconds.

Configure SSH
Two files need to be edited in order to enable two-factor authentication in SSH.

vim /etc/pam.d/sshd

Add this line:

auth required nullok

Where to put this in the file? That depends. When you put it at the top, SSH will first ask a verification code, then a password. To me this sounds unlogical, so I placed it just below this line:

@include common-auth

The ‘nullok’ option, by the way, tells PAM whenever no config for 2-factor authentication is found, it should just ignore it. If you want SSH logins to fail, when no two-factor authentication is configured, you can delete the option. Be warned to at least config it for one user, or you will be locked out of your server.

Now tel SSH to ask for the verification code:

vim /etc/ssh/sshd_config

Edit the setting, it’s probably set to ‘no’:

ChallengeResponseAuthentication yes

Now all you need to do is restart SSH. Keep a spare SSH session logged-in, just in case.

/etc/init.d/ssh restart
SSH 2-factor authentication in action

SSH two-factor authentication in action

SSH will now ask for a verification code when you do an interactive login. When a certificate is used, no verification code is asked.

Configuring Ubuntu Desktop
The Ubuntu desktop can also ask for a verification code. Just edit one file:

vim /etc/pam.d/lightdm

And add this line, just like SSH:

auth required nullok

The login screen should now ask for a verification code. It looks like:

Two-factor authentication in Ubuntu

Two-factor authentication in Ubuntu

You can setup any PAM service like this, the basic principles are all the same.

Skip two-factor authentication if logging in from the local network
At first this is all very cool, but soon it becomes a bit annoying, too. When I SSH from a local network, I just don’t want to enter the verification code because I trust my local network. When I SSH from remote, a verification code is required. One way to arrange that, is always login with certificates. But there is another way to configure it: using the pam_access module. Try this config:

auth [success=1 default=ignore] accessfile=/etc/security/access-local.conf
auth required nullok

The config file, /etc/security/access-local.conf looks like:

# Two-factor can be skipped on local network
 + : ALL :
 + : ALL : LOCAL
 - : ALL : ALL

Local login attempts from will not require two-factor authentication, while all others do.

Two-factor only for users of a certain group
You can even enable two-factor authentication only for users of a certain group. Say, you’d want only users in the ‘sudo’ group to do use two-factor authentication. You could use:

auth [success=1 default=ignore] user notingroup sudo
auth required nullok

By the way, “[success=1 default=ignore]” in PAM means: on success, skip the next 1 auth provider (two-factor authentication), on failure, just pretend it didn’t happen.

Moving the config outside the user’s home
Some people prefer not to store the authentication data in the user’s home directory. You can do that using the ‘secret’ parameter:

mkdir -p /var/lib/google-authenticator/
mv ~remibergsma/.google_authenticator /var/lib/google-authenticator/remibergsma
chown root.root /var/lib/google-authenticator/*

The PAM recipe will then become:

auth required nullok user=root secret=/var/lib/google-authenticator/${USER}

As you’ve seen, PAM is very flexible. You can write almost anything you want.

Try the Google Authenticator two-factor authentication. It’s free and adds another security layer to your home server.

Since I started my new job last week, I no longer work on a Mac desktop. I now use Ubuntu instead. Our company has an Exchange server and uses Active Directory for authentication. Using a Linux desktop in a Windows environment can be a bit challenging. While you can use Thunderbird to use imap, features like calandar and global address book are not available. Of course there’s Terminal Services using RDP but this is not ideal. My new colleagues pointed me to a little tool called DavMail.

davmail-settingsDavMail is a Java program that makes an imap, pop, smtp, calandar and even ldap server available to localhost. It translates requests to the local server and points them to the OWA (Outlook Web Access url) provided by Exchange. This way, my Linux Desktop can use all the services normally not available on Linux. Mail (both incoming and outgoing), calandar and even auto completion of names and e-mail addresses of all of my colleagues work perfectly.

This rocks, thanks guys!

I’m used to my Mac where the direction of scrolling was changed to a more ‘natural’ way starting with OSX Lion. It’s the same way scrolling works on mobile devices. When working on Ubuntu, I want to have the same behaviour because I’m used to it.

To do so you need to edit (or create) the .Xmodmap file in your homedir.

This is the default:

cat ~/.Xmodmap
pointer = 1 2 3 4 5

I changed it to:

vim ~/.Xmodmap
pointer = 1 2 3 5 4

This essentially makes the last two (scroll wheel) buttons work the other way around, and that is exactly the behaviour I want.

To enable this setting you have to either logout (and back in) of your X environment, or run this oneliner (Thanks James!):

xmodmap -e "pointer = 1 2 3 5 4"

By editing both the Xmodmap file and running the above oneliner, the new setting works immediately and also keeps working when you (for example) reboot your machine.