Archives For networking

It’s been one year after Cosmic started, forked off of the Apache CloudStack project. Cosmic 5.3 brings easy-to-use connectivity between VPCs. CloudStack users will recognise the Private Gateway functionality, which has been completely refactored in Cosmic 5.3. This blog shows what’s new.

The goal: connect VMs together that live in different VPCs
Basically, what you want is to setup a bunch of VPCs, create networks inside them where the VMs will live. Once you do that, there is no easy way to connect VMs from one VPC to another. Sure, you could make a VPN connection, and that is preferred in some cases. When you work in the same availability zone, and for example want Management VMs to access Acceptance or Production, there’s an easy alternative to get connectivity.

Create a Private Network
In Cosmic 5.3 you can create a network of type Private. A default network offering for this is created automatically, the only real difference is setting the type to Private (other networks are usually of type Isolated).


Create Private Network

If you want, you can specify a VLAN of LSwitch manually by selecting an offering that supports it. You’d only do that in special cases, as it requires manual setup that is otherwise done automatically for you.


Create Private Network with a specific VLAN

Connect a VPC to the Private Network
Next, we’ll want to connect a VPC to this Private Network. That is done using a Private Gateway.


Creating a Private Gateway


Private Gateway is connected to the Private Network

Now, the VPC is connected to the Private Network. You will want to repeat this step for all VPCs you want to connect. In my case I added 3 VPCs with each one Private Gateway. Make sure they have an unique ip address on the Private Network.


Overview of VPCs created

VPC1 has CIDR of and has a Private Gateway using connected to the Private Network. VPC2 has and VPC3 has as its ip address on the Private Network.

Making the VPCs aware of each other
By setting static routes to the other VPCs, to their ip address on the Private Network, it becomes easy to route traffic.


Setting routes to the other VPCs

VPC1 now knows that is reachable via (VPC2) and is reachable via (VPC3). If you fill the route table for VPC2 and VPC3 in the same way (create routes to the CIDRs of the other VPCs) then all VMS in these three VPCs can reach each other!


Three VMS in their own VPC


Inter-VPC traffic over the Private Network

Best of all, you can create this as a Domain Admin user.

Improvements over Apache CloudStack:

  • A special network type Private was created, to make a clear distinction between types Guest, Public and Private (lots of hacks in the code removed)
  • Able to use Domain Admin user, instead of ROOT user (even the RBAC in CloudStack cannot delegate the creation of Private Gateways to non-ROOT users).
  • Add a Private Gateway without pre-setting up a VLAN or Lswitch
  • Able to set Static Routes in the Route Table on the VPC level, rather than on the Private Gateway level. This allows setting routes to anything the kernel of the router VM accepts which is very flexible.

Cosmic 5.3 has made connecting VPCs together a whole lot easier. The fact that Domain Admins can now set this up (using Terraform & friends) makes it an easy to consume feature. Combined with the flexible static routes, one can basically implement any network design.

Get Cosmic for free now!

Here are the slides from my keynote presentation at the CloudStack Collaboration Conference in Dublin today.

Tomorrow I’ll do another presentation and talk about automation cloud operations!

When you boot your Raspberry Pi with Raspbian, it will get its ip address from a DHCP server. Most tutorials I’ve seen, ask you to login to a ‘router’ (that serves as DHCP server as well) to find the assigned ip address. When you haven’t access to the DHCP server, how do you find the Pi’s ip address?
You should know the MAC address of a Raspberry Pi always has a ‘b8:27:eb’ prefix. Using a Linux utility called ‘arp-scan‘ you can discover MAC addresses and their corresponding ip addresses. Arp-scan sends ARP packets to hosts on the local network and displays any responses that are received.
First, install arp-scan:
apt-get install arp-scan
Then locate all Raspberry Pi’s on the local network using this command:
arp-scan --interface=eth0 --localnet | grep b8:27:eb
You will need to be root, or arp-scan must be SUID root, in order to run arp-scan, because the functions that it uses to read and write  packets require root privilege.
Result: b8:27:eb:00:11:ab (Unknown)
Using the ip address, it is now possible to SSH into your Raspberry Pi:
ssh -l pi

OpenVPN is a very powerful tool to connect to a remote network in a secure and easy way. But while it’s quite easy to set it up (see my post about OpenVPN on Raspberry Pi), getting all the pieces together takes some more network understanding and configuration. I created an image to show what’s going on when a OpenVPN client connects to the server.

OpenVPN networking

The grey network is the local network you’re connecting to, the green network is created by OpenVPN. After a successful connection, the OpenVPN server can ‘push’ a route to the OpenVPN client to make it aware of the grey network that is available through the OpenVPN connection. In this case you’d add this setting to the OpenVPN server:

push "route"

Now that the OpenVPN client knows how to find the Linux server on the grey network, it should be working, right? Nope. The Linux server does not know about the green network. So, when a OpenVPN client connects with a 10.8.0.x ip-address, this server does not know where to send the response to. When this happens, the Linux server sends its traffic to the default gateway of the local network: the router of the grey network.

But unfortunately, the router does not know about the green network neither. Since 10.8.0.x is a non-routable address (RFC 1918), it drops the packets because it has no way of knowing where to find this private network. To the OpenVPN client, this may look like there’s something wrong with the VPN connection, because even a simple ping will not work and times out. In reality, this is a routing problem.

The solution? Make the Linux server or at least the router aware of the green network, and tell them where to send traffic to this network. This is done by adding a route like this:

route add -net gw

Assuming the local network ip-address of the OpenVPN server is

This way the hosts in the grey network know about the green network and send their traffic to the OpenVPN server. Of course the OpenVPN server knows how to reach the OpenVPN client and all is working!

You can extend this further and add routes to the network of the VPN client and make hosts on both networks communicate to each other. This is useful for example to connect a branch-office to the corporate network. I’ll write about this in more detail some other time.

Hopefully this post brings some insight in what goes on when using networks and VPN connections.

I’ve been working a lot with CloudStack Advanced networking and find it very flexible. Recently, I had another opportunity to test its flexibility when a customer called: “We want VM’s in your CloudStack cloud, but these VM’s are only allowed be reachable from our office, and not from the public internet”. Firewalling? No, they required us to use their VPN solution.

Is CloudStack flexible enough for this to work? Yes, it is. In this blog I’ll tell you how we did it. And it doesn’t even matter what VPN device you use. This will work regardless of brand and features, as lang as it supports a public ip-address to connect over the internet to another VPN device, and has a private network behind it. All VPN devices I know of support these basic features.

VPN (Virtual Private Networking)
The client’s office is connected to the internet and has a VPN device. We received another device as well to host in our data center and the two talk to each other over the public internet in a secure way. Probably speaking IPsec or similar but that is beyond the scope of this blog.

The VPN device in the data center has a public ip-address on its WAN port but also has some ports for the internal network. We configured it to use the same network CIDR as we did in the CloudStack network we created for this customer. Let’s use as an example in this blog. And now the problem: this cloud network is a tagged network and the VPN device we received is not VLAN-capable.

VLANs in the Cloud
CloudStack Advanded networking relies on VLANs. Every VLAN has its own unique ID. Switches use this VLAN ID to keep the VLAN networks apart and make sure they’re isolated from each other. Most switches support VLANs as well, and that’s were we’ll find the solution to this problem.

Configuring the switch

We connected the VPN device to our switch and set its port to UNTAGGED for the VLAN ID the CloudStack network uses. In other words, devices connected to this port now do not need to know about the VLAN. The switch will add it as traffic flows. This means the VPN device will use an ip-address in the range and is able to communicate with the VM’s in the same network. The CloudStack compute nodes have their switch ports set to TAGGED and the switch makes communication between them possible.

Overview of ip-addresses:

  • – the internal VPN device ip-address in the data center
  • – the first VM’s ip-address
  • – the second VM’s ip-address

The VM’s have their default gateway set to the VPN device’s address. Also, the office needs to be configured in a way it knows the network is handled by the VPN device located there. Users in the office will now be able to access the VM’s on the network.

While the VM’s are hosted on our CloudStack cloud on the internet, they do not have public ip-addresses and thus are not reachable. The only public ip-address for this customer is the one configured on the VPN device in the data center. This provides the same level of security as you’d have with physical servers but adds the power of a cloud solution.

Thanks to the flexibility of the CloudStack Advanced Networking this cloud be done!

When migrating an ip-address to another server, you will notice it will take anywhere between 1 and 15 minutes for the ip-address to work on the new server. This is caused by the arp cache of the switch or gateway on the network. But don’t worry: you don’t just have to wait for it to expire.

Why it happens
ARP (Address Resolution Protocol) provides a translation between ip-addresses and mac-addresses. Since the new server has another mac-address and the old one stays in the cache for some time, connections will not yet work. The cache usually only exists for some minutes and prevents asking for the mac-address of a certain ip-address over and over again.

One solution to this problem is to send a command to the gateway to tell it to update its cached mac-address. You need the ‘arping’ utility for this.

Installing arping
There are two packages in Debian that contain arping:

arping - sends IP and/or ARP pings (to the mac address)
iputils-arping - Tool to send ICMP echo requests to an ARP address

I’ve had best results with the ‘iputils’ one, so I recommend to install that one. This is mainly because the other package’s command does not implement the required -U flag.

aptitude install iputils-arping

I haven’t installed arping on CentOS yet, but was told the package is in the RPMForge repository.

Using arping
The command looks like this:

arping -s ip_address -c1 -U ip_addresss_of_gateway

-s is the source ip-address, the one you want to update the mac-address of
-c1 sends just one arping
-U is Unsolicited arp mode to update neighbours’ arp caches
This is followed by the ip-address of the gateway you want to update. In most cases this is your default gateway for this network.

Example: you moved to a new server and your gateway is, you’d run:

arping -s -c1 -U

After you’ve send the arping, the gateway will update the mac-address it knows of the ip-address and traffic for this ip-address will start flowing to the new server.

Bottom line: whenever you migrate an ip-address to another server, use arping to minimize downtime.

Back in March I wrote a blog on how to create a network without a Virtual Router.  I received a lot of questions about it. It’s also a question that pops up now and then on the CloudStack forums. In the meanwhile I’ve worked hard to implement this setup at work. In this blog I’ll describe the concept of working with a CloudStack setup that has no Virtual Router.

First some background. In Advanced Networking, VLAN’s are used for isolation. This way, multiple separated networks can exist over the same wire. More about VLAN technology in general on this wikipedia page. For VLAN’s to work, you need to configure your switch so it knows about the VLAN you use. VLAN’s have an unique id between 1 and 4096. CloudStack configures this all automatically, except for the switch. Communication between Virtual Machines in the same CloudStack network (aka VLAN) is done using the corresponding VLAN-id. This all works out-of-the-box.

It took me some time to realize how powerful this actually is. One can now combine both VM’s and physical servers in the same network, by using the same VLAN for both. Think about it for a moment. You’re now able to replace the Virtual Router with a Linux router simply by having it join the same VLAN(s) and using the Linux routing tools.

Time for an example. Say we have a CloudStack network using VLAN-id 1234, and this network is created without a Virtual Router (see instructions here). Make sure you have at least 2 VM’s deployed and make sure they’re able to talk to each other over this network. Don’t forget to configure your switch. If both VM’s are on the same compute node, networking between the VM’s works, but you won’t be able to reach the Linux router later on if the switch doesn’t know the VLAN-id.

Have a separate physical server available running Linux and connect it to the same physical network as your compute nodes are connected to. Make sure the ip’s used here are private addresses. In this example I use:


The Linux router needs two network interfaces: one to the public internet (eth0 for example) and one to the internal network, where it connects to the compute nodes (say eth1). The eth1 interface on the router has ip-address and it should be able to ping the compute node(s). When this works, add a VLAN interface on the router called eth1.1234 (where 1234 is the VLAN-id CloudStack uses). Like this:

ifconfig eth1.1234 up

Make sure you use the correct ip-address range and netmask. They should match the ones CloudStack uses for the network. Also, note the ‘.’ between the eth1 and the VLAN-id. Don’t confuse this with ‘:’ which just adds an alias ip.

To check if the VLAN was added, run:

cat /proc/net/vlan/eth1.1234

It should return something like this:

eth1.1234 VID: 1234 REORDER_HDR: 1 dev->priv_flags: 1
 total frames received 14517733268
 total bytes received 8891809451162
 Broadcast/Multicast Rcvd 264737
 total frames transmitted 6922695522
 total bytes transmitted 1927515823138
 total headroom inc 0
 total encap on xmit 0
Device: eth1
INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
 EGRESS priority mappings:

Tip: if this command does not work, make sure the VLAN software is installed. In Debian you’d simply run:

apt-get install vlan

Another check:

ifconfig eth1.1234

It should return something like this:

eth1.1234 Link encap:Ethernet HWaddr 00:15:16:66:36:ee 
 inet addr: Bcast: Mask:
 inet6 addr: fe80::215:17ff:fe69:b63e/64 Scope:Link
 RX packets:14518848183 errors:0 dropped:0 overruns:0 frame:0
 TX packets:6925460628 errors:0 dropped:15 overruns:0 carrier:0
 collisions:0 txqueuelen:0 
 RX bytes:8892566186128 (8.0 TiB) TX bytes:1927937684747 (1.7 TiB)

Now, the most interesting tests: ping vm1 and vm2 from the linux router, and vice versa. It should work, because they are all using the same VLAN-id. Isn’t this cool? You just connected a physical server to a virtual one! 🙂

You now have two options to go from here:

1. Use a LoadBalancer (like Keepalived) and keep the ip’s on the VLAN private using Keepalived’s NAT routing. The configuration is exactly the same as if you had all physical servers or all virtual servers.

2. Directly route public ip’s to the VM’s. This is the most interesting one to explain a bit further. In the example above we’ve used private ip’s for the VLAN. Imagine you’d use public ip addresses instead. For example:

router1: (eth1.1234; eth1 itself remains private)

This also works: vm1, vm2 and router1 are now able to ping each other. A few more things need to be done on the Linux router to allow it to route the traffic:

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/eth1/proxy_arp

Finally, on vm1 and vm2, set the default gateway to router1; in this example.

How does this work? The Linux router also answers arp requests for the ip’s in the VLAN. Whenever traffic comes by for vm1, router1 answers the arp request and routes the traffic over the VLAN to vm1. When you’d run a traceroute, you’ll see the Linux router appear as well. Of course you need to have a subnet of routable public ip’s assigned by your provider for this to work properly.

To me this setup has two major advantages:

1. No wasted resources for Virtual Routers (one for each network)
2. Public ip’s can be assigned directly to VM’s; you can even assign multiple if you like.

The drawbacks? Well, this is not officially supported nor documented. And since you are not using the Virtual Router, you’ll have to implement a lot of services on your own that were normally provided by the Virtual Router. Also, deploying VM’s in a network like this only works using the API. To me these are all challenges that make my job more interesting 😉

I’ve implemented this in production at work and we successfully run over 25 networks like this with about 100-125 VM’s. It was a lot of work to configure it all properly and to come up with a working solution. Now that it is live, I’m really happy with it!

I realize this is not a complete step-by-step howto. But I do hope this blog will serve as inspiration for others to come up with great solutions build on top of the awesome CloudStack software. Please let me know in the comments what you’ve come up with! Also, feel free to ask questions: I’ll do my best to give you some directions.