It’s been one year after Cosmic started, forked off of the Apache CloudStack project. Cosmic 5.3 brings easy-to-use connectivity between VPCs. CloudStack users will recognise the Private Gateway functionality, which has been completely refactored in Cosmic 5.3. This blog shows what’s new.
The goal: connect VMs together that live in different VPCs Basically, what you want is to setup a bunch of VPCs, create networks inside them where the VMs will live. Once you do that, there is no easy way to connect VMs from one VPC to another. Sure, you could make a VPN connection, and that is preferred in some cases. When you work in the same availability zone, and for example want Management VMs to access Acceptance or Production, there’s an easy alternative to get connectivity.
Create a Private Network In Cosmic 5.3 you can create a network of type Private. A default network offering for this is created automatically, the only real difference is setting the type to Private (other networks are usually of type Isolated).
Create Private Network
If you want, you can specify a VLAN of LSwitch manually by selecting an offering that supports it. You’d only do that in special cases, as it requires manual setup that is otherwise done automatically for you.
Create Private Network with a specific VLAN
Connect a VPC to the Private Network Next, we’ll want to connect a VPC to this Private Network. That is done using a Private Gateway.
Creating a Private GatewayPrivate Gateway is connected to the Private Network
Now, the VPC is connected to the Private Network. You will want to repeat this step for all VPCs you want to connect. In my case I added 3 VPCs with each one Private Gateway. Make sure they have an unique ip address on the Private Network.
Overview of VPCs created
VPC1 has CIDR of 10.1.0.0/16 and has a Private Gateway using 172.16.0.1 connected to the Private Network. VPC2 has 172.16.0.2 and VPC3 has 172.16.0.3 as its ip address on the Private Network.
Making the VPCs aware of each other By setting static routes to the other VPCs, to their ip address on the Private Network, it becomes easy to route traffic.
Setting routes to the other VPCs
VPC1 now knows that 10.2.0.0/16 is reachable via 172.16.0.2 (VPC2) and 10.3.0.0/16 is reachable via 172.16.0.3 (VPC3). If you fill the route table for VPC2 and VPC3 in the same way (create routes to the CIDRs of the other VPCs) then all VMS in these three VPCs can reach each other!
Three VMS in their own VPCInter-VPC traffic over the Private Network
Best of all, you can create this as a Domain Admin user.
Improvements over Apache CloudStack:
A special network type Private was created, to make a clear distinction between types Guest, Public and Private (lots of hacks in the code removed)
Able to use Domain Admin user, instead of ROOT user (even the RBAC in CloudStack cannot delegate the creation of Private Gateways to non-ROOT users).
Add a Private Gateway without pre-setting up a VLAN or Lswitch
Able to set Static Routes in the Route Table on the VPC level, rather than on the Private Gateway level. This allows setting routes to anything the kernel of the router VM accepts which is very flexible.
Conclusion Cosmic 5.3 has made connecting VPCs together a whole lot easier. The fact that Domain Admins can now set this up (using Terraform & friends) makes it an easy to consume feature. Combined with the flexible static routes, one can basically implement any network design.
I was the Release Manager for Apache CloudStack versions 4.6, 4.7 and 4.8 and during this time many people told me they thought it’s hard to build and package Apache CloudStack. The truth is, it’s not that hard once you know the steps to take. 😉
Next to that there’s the release pace. You may want to move in a different pace than the project is. There’ve been lots of discussions for example on a fast release cycle, and also on LTS-releases on the other hand. In this blog post I’ll show you how easy it is to create your own CloudStack release to satisfy the demands of your organisation.
Maven Compiling Apache CloudStack is done using Maven. You need to install this tool in order to work with releases so let’s do it. Let’s assume a CentOS 7 box you want to compile this on. It should pretty much work on any OS.
We also install Java and some tools needed to compile Apache CloudStack.
Versioning When you’re building your own release of Apache CloudStack, you have two options:
Rebuild an existing version
Create a new version
You’ve to keep in mind that when you create a new version, you need also to create so-called upgrade-paths: how the database can be upgraded to your version. When you choose option 1, and rebuild an existing version, this is not necessary. This sounds easy, but on the other hand, it’s confusing as there’s no way to tell the difference later on.
Apache CloudStack works with versions like 4.8.0 and 4.7.1, etc. The upgrade mechanism will only consider the first 3 parts. This means, we are free to create 4.8.0.16 (our 16th custom version of 4.8.0) as long as we do not touch the database. That sounds like a nice way to make custom daily releases, and at the same time can be used for those wanting to build LTS (Long Term Support) versions.
Setting the version The question then is, how can we modify the version of Apache CloudStack? Well, there’s actually a tool supplied in the source that does this for you. It’s called setnextversion.sh. Here’s how it works:
usage: ./tools/build/setnextversion.sh -v version [-b branch] [-s source dir] [-h]
-v sets the version
-b sets the branch (defaults to 'master')
-s sets the source directory
-h
To build our custom 4.8.0.16 version off the 4.8 branch we run:
found 4.8.0 setting version numbers
[master 27fa04a] Updating pom.xml version numbers for release 4.8.0.16
126 files changed, 130 insertions(+), 130 deletions(-)
committed as 858805957e2c7e0a0fbeb170a7f1681a73b4fb7a
The result is a new commit that changed the versions in the POMs. You an see it here:
git log
You’ve successfully set the version!
Compiling the custom version Let’s compile it. This works like with any release build:
mvn clean install -P systemvm
This will take a few minutes. You should see your new version flying by lots of times.
After a few minutes the build completes.
By the way, if you want to skip the unit tests to speed up the process, add -DskipTests to the mvn command.
RPM packages The source also contains scripts to build packages. To build packages for CentOS 7 for example, do this:
cd ./packaging
./package.sh -d centos7
You should see this:
Preparing to package Apache CloudStack 4.8.0.16
When the script finishes, you can find the RPM packages here:
ls -la ../dist/rpmbuild/RPMS/x86_64/
Installing the new version You can either use the war that is the result of the compile, or install the generated RPM packages. Installing and upgrading is out-of-scope in this blog, so I assume you know how to install a stock version of CloudStack. I first installed a stock 4.8.0 version, then (as shown above) built 4.8.0.16 and will show the upgrade.
As soon as you start the management server with the new version you will see this:
2016-03-14 12:34:09,986 DEBUG [c.c.u.d.VersionDaoImpl] (main:null) (logid:) Checking to see if the database is at a version before it was the version table is created
2016-03-14 12:34:09,995 INFO[c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version = 4.8.0 Code Version = 4.8.0.16
2016-03-14 12:34:09,995 INFO[c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version and code version matches so no upgrade needed.
This is actually very nice. We made a custom version and CloudStack still assumes it’s 4.8.0 and so no upgrade of the data base is needed. This obviously means that you cannot do this when your patch requires a data base change.
When we look at the database, we can confirm we still run a 4.8.0-compatible release:
From this table, one cannot tell we upgraded to our custom version. But when you look closer, the new version is active.
This is the version an agent reports:
Also, the UI will show the custom version in the About box. This way users can easily tell what version they are running.
Conclusion Creating your own custom version of Apache CloudStack may sound complicated but we’ve seen it’s pretty easy to do so. Creating a custom release will provide you with a lot of flexibility, especially if you combine it with war dropping. By numbering your version like 4.8.0.x you don’t have to worry about upgrade paths.
Upgrading CloudStack can be tough at times. Some say therefore you should do as little upgrades as possible. My approach is to make it easy and make the number of changes as small as possible. The more upgrades, the better. The smallest unit is a single Pull Request with a single commit. That we should be able to deploy to production in an automated way with almost no effort. At Schuberg Philis, our current deploy record to our Mission Critical Clouds is five times a day. Here’s how we do it.
CloudStack is a Java application
CloudStack is a java application and when you compile it, it creates jars and also a war. A war-file (short for Web application ARchive) according to Wikipedia, is used “to distribute a collection of JavaServer Pages, Java Servlets, Java classes, XML files, tag libraries, static web pages (HTML and related files) and other resources that together constitute a web application.” All-in-one, that’s handy! Now, Tomcat is used to serve Java applications and one can add an application to it by what we call “dropping a war“. This means that Tomcat will unpack and start the application inside the war-file.
Our goal: we compile a war from the CloudStack source and then tell Tomcat to run it. Let’s me show you how we do this.
Installing Tomcat
In this blog I’ll show you how to do it on CentOS 7, as this is currently my favourite distribution. Tomcat runs on any OS, so with slight adjustments you can use another distribution as well.
yum install tomcat mkisofs
The package mkisofs is used by the Management Server. We’ll configure Tomcat later on.
The MySQL connector story
If you try dropping a CloudStack war in Tomcat, you will find that it doesn’t work because the MySQL driver cannot be found. My colleague Miguel Ferreira sent a Pull Request to add it but as it seems that wasn’t allowed due to licensing issues.
There’s a reason why the MySQL connector is not a dependency – it is Cat-X licensed, which means we may not depend on it in the default build.
Miguel then investigated alternatives and finally found that when you install the mysql-connector-java RPM package, and then put it at the front of the ClassPath, Tomcat will pick it up automatically. Details are in the Pull Request. Awesome work dude!
Then go into the root of the source folder and run:
mvn clean install -P systemvm
If you have a fast computer, you may add -T 4 to compile with 4 treads. It’s faster, that’s all.
After a while you see:
[INFO] --------------------------------------------
[INFO] BUILD SUCCESS
[INFO] --------------------------------------------
[INFO] Total time: 06:09 min (Wall Clock)
[INFO] Finished at: 2016-02-29T11:21:50+01:00
[INFO] Final Memory: 109M/1571M
[INFO] --------------------------------------------
Notice this line:
[INFO] Building war: /data/git/cs1/cloudstack/dist/rpmbuild/BUILD/cloudstack-4.8.0/client/target/cloud-client-ui-4.8.0.war
So, the war is in client/target/ folder.
Copy the war-file to the Management server that will run Tomcat.
You can build the war-file on the management server itself, on another server, even on your laptop if you like. Just make sure you copy it to the management server (mine is named cs1).
The war-file needs to be added to Tomcat. You can use the UI manager for this and do it while Tomcat keeps running. To keep this demo easy, let’s stop Tomcat and assume it’s the only application it runs:
systemctl stop tomcat
Be sure to go to the Tomcat home folder:
cd ~tomcat/webapps/
Then copy the war-file you built, but rename it to ‘client.war‘. This is because the application is supposed to be called ‘client‘. This is why the url also has ‘/client‘.
[root@cs1 webapps]# ls -al
total 247836
drwxrwxr-x. 3 root tomcat 4096 Feb 29 10:42 .
drwxr-xr-x. 3 root tomcat 4096 Oct 28 12:49 ..
-rw-r--r--. 1 root root 253766159 Feb 29 10:41 client.war
Starting Tomcat will automatically unpack the war and start it:
systemctl start tomcat
That’s it! Now, have a look at the folder:
[root@cs1 webapps]# ls -al
total 247836
drwxrwxr-x. 3 root tomcat 4096 Feb 29 10:42 .
drwxr-xr-x. 3 root tomcat 4096 Oct 28 12:49 ..
drwxr-xr-x. 11 tomcat tomcat 4096 Feb 29 10:42 client
-rw-r--r--. 1 root root 253766159 Feb 29 10:41 client.war
Management server log is here:
less ~tomcat/vmops.log
As you’ll see, CloudStack is started and the UI is available after a few minutes.
Missing files in war-file
If you want to deploy a new cloud, you’ll find that some stuff goes wrong. First of all, the database scripts are not in the war. You can place them in the Tomcat home dir.
To generate an archive of the files from the CloudStack source:
cd setup/db; tar -c -f ~/Downloads/db-scripts-${CLOUD_VERSION}.tar.gz -pPz db/; cd -
Copy this to the Tomcat homedir, and extract.
cd ~tomcat
tar zvxf db-scripts-${CLOUD_VERSION}.tar.gz
chown tomcat.tomcat ./db -R
The same counts for any additional scripts, like ‘cloudstack-setup-management‘ etc. They are not part of the war so you need to copy them should you want to use them.
Legacy /etc/cloudstack folder
If you’re looking for config files, they’re all inside the webapps folder. This means both a new location and that you need to backup/restore when dropping a new war-file.
You could make a symlink to the new location like this:
mkdir -p /etc/cloudstack
cd /etc/cloudstack
ln -s /var/lib/tomcat/webapps/client/WEB-INF/classes management
Another option is to copy the files and add ‘/etc/cloudstack/management‘ to the CLASSPATH. The benefit is that you keep your config files out of the client folder so persistence is easier.
The user tomcat runs as
CloudStack wants it to run as user ‘cloud‘, don’t ask me why. For development it works fine as user tomcat, but for some stuff to work (like encryption) you better change it to user ‘cloud‘.
vim /etc/sysconfig/tomcat
TOMCAT_USER="cloud"
Automation!
We obviously want automation in place for this to work without even thinking about it. You can use your favourite configuration management tool.
Flipping an attribute does the trick.
The work-flow we use looks like this:
– Build new war, then upload to some central place – Flip a Chef attribute to indicate the new version should be installed – Chef then takes over and:
– downloads the war-file – downloads the db-scripts.tar.gz – stops Tomcat – backups settings, like db.properties and such – removes old files – puts new client.war in ~tomcat/webapps folder – extracts db-scripts.tar.gz in Tomcat home folder – starts Tomcat
Loadbalanced Management Servers
As you can see, there is some downtime involved when upgrading to a newer version. When you setup two or more CloudStack application servers with a loadbalancer in front of them, you can upgrade the management servers one-by-one without service interruption.
Conclusion
CloudStack clearly wasn’t designed for easy war-dropping into Tomcat, but as we’ve seen it can be done. Once you overcome a few hurdles, you’ll find it is actually much easier compared to upgrading RPMs all the time. Especially when you want to deploy the same version, with an extra bugfix this comes in handy.
It’d be nice to enhance CloudStack to fully support war drops. Even though users may want to use RPM packages we could simplify them by getting a war artifact (signed) from Maven Central or such, and then have the RPM package setup the things around it. We then would get the best of both worlds: Use RPMs if you want, be able to simply drop a war-file if you want to go faster.
For Schuberg Philis, this allows us to do continuous deployments to our production clouds. By making the numer of changes as smal as possible, we have many small deployments that are easy to test and easy to revert, should we have to.
At Schuberg Philis, we manage quite a farm of XenServer clusters. As the number of clusters we operate goes up, so goes the time it takes to manage them all. And, let’s be honest: patching and upgrading isn’t the most exciting work. We needed more automation and I took the challenge 😉
The challenge Patching itself is not a challenge, as we automated that already using Chef. The challenge comes when maintenance tasks require a hypervisor reboot in order to take effect. In such cases, some magic is needed to keep all VMs running, because we aim for zero down time due to maintenance. This blog describes how I automated rebooting full XenServer clusters while all VMs keep running.
N+1 concept Our clusters are designed based on N+1, which means that if we have a cluster of 6 XenServer hypervisors, we use ~83% (5/6) of its capacity. In this case, when one hypervisor crashes, we have still room for the VMs on the crashed hypervisor, to start again on one of the remaining hosts.
When executing maintenance on our hypervisors, we also use this concep. In this case, we make one hypervisor empty by live migrating all its VMs to the other hypervisors. Now, without any VMs left on it, we can easily patch or upgrade that hypervisor without impact.
Goal The goal that I had in mind was that I wanted to be able to run a one-liner to automatically reboot a XenServer cluster. Based on the N+1 principle above, that should be possible. The image below shows the process:
This is what happens:
Set the specified cluster to unmanage in CloudStack
Turn OFF XenServer poolHA for the specified cluster
For any hypervisor it will do this (poolmaster first):
put it to Disabled aka Maintenance in XenServer
live migrate all VMs off of it using XenServer evacuate command
when empty, it will reboot the hypervisor
will wait for it to come back online (checks SSH connection)
set the hypervisor to Enabled in XenServer
continues to the next hypervisor
When the rebooting is done, it enables XenServer poolHA again for the specified cluster
Finally, it sets the specified cluster to Managed again in CloudStack
CloudStack will update its admin according to the new situation Then the reboot cyclus for the specified cluster is done!
About CloudStack and XenServer Since XenServer has the concept of a pool (you could call it a cluster) and one of the hosts is the pool master (it distributes the work to the other cluster members), it is important to know that we need to start with the pool master.
If you want to be sure CloudStack is not touching the cluster while you are performing maintenance, you can put it to an ‘Unmanaged state‘. In the old days we had to do this to prevent CloudStack from electing a new pool master while the previous one was rebooting. As it now relies on XenHA to handle electing a new pool master, I think it’s not needed per se. I just kept it in the scripts to be sure.
CloudStackOps The script I wrote for this is called ‘xenserver_rolling_reboot.py‘ and is part of the CloudStackOps Toolbox. If you haven’t heard of it before, please check this video of a recent talk I did.
As we saw, the script live migrates all VMs off of the pool master, then reboots it. After that, it does the same with all the other hosts. One-by-one. Until they are all rebooted. All VMs stay online during the maintenance.
About live migrations You would think that it’d be easiest to have CloudStack do the live migrations, and I thought that too. The problem is, that CloudStack live migrates the VMs to a random hypervisor in the same cluster. It depends a bit on the deployment planner but I found little difference in this case. Why is that a problem? Well, you could end up ‘pushing VMs forward’ all the time. I want to live migrate a VM as less times as possible. This makes the whole process a lot faster. Let’s have a look on how the live migrations should work in my opinion:
Conclusion: VMs on the pool master will be live migrated twice. All others only once. Optionally we could rebalance the cluster, causing more live migrations. For now, I kept it to the minimum as I’m OK with some hosts being almost empty.
In order to control the live migrations shown in the images above, I decided to talk to Xapi directly instead of to CloudStack.
Speeding up live-migrations Unfortunately the whole process was quite slow when I first tested it. The first idea was to use ‘xe host-evacuate‘ but this script can only do one live migration at a time. I asked some XenServer folks about it. Conclusion is that it cannot be done at this time.
Well, that means I’d to come up with something of my own to speed up the live migrations. To prove it would be possible to migrate faster, we used XenCenter and clicked around. We could easily do 5-10 threads and it was a lot faster. But who wants to click these days? Not me.
After some searches on Google I came up with a very simple solution: use the -P flag of xargs. I wrote a small script that looks up the VMs running on the hypervisor you run the script and it will then calculate a migration plan according to what we discussed above. Then, I’d have a list of ‘xe vm-migrate‘ commands that I run in parallel using xargs. Very simple, very effective.
By using 5 threads, I got twice the speed so that saved a lot of waiting time. I also tried more threads, but it didn’t go any faster. The ‘xenserver_rolling_reboot.py‘ script has a –threads flag, so feel free to play with it. It defaults to 5 threads.
Verified on Production My team uses this script to do automated patching and rebooting. For the patching itself we use Chef, but that’s the easy part. Rebooting the clusters is what takes time. Now that it is automated, we only have to keep an eye on it. We used it in production to reboot hundreds of XenServer hypervisors without issues.
Conclusion Automation is key in today’s cloudy environments. This blog shows how I automated the regular maintenance of our XenServer farm. Patching and rebooting is now easy!
Today I hosted a CloudStack hands-on workshop at DevOpsDays Amsterdam, together with my colleague Fred Neubauer. It was really awesome to see all attendees building their own CloudStack cloud. And succeeding!
I was the 
Upgrading 
CloudStackOps
Today I hosted a CloudStack hands-on workshop at 
