Remi Bergsma's blog

Upgrading CloudStack can be tough at times. Some say therefore you should do as little upgrades as possible. My approach is to make it easy and make the number of changes as small as possible. The more upgrades, the better. The smallest unit is a single Pull Request with a single commit. That we should be able to deploy to production in an automated way with almost no effort. At Schuberg Philis, our current deploy record to our Mission Critical Clouds is five times a day. Here’s how we do it.

CloudStack is a Java application

CloudStack is a java application and when you compile it, it creates jars and also a war. A war-file (short for Web application ARchive) according to Wikipedia, is used “to distribute a collection of JavaServer Pages, Java Servlets, Java classes, XML files, tag libraries, static web pages (HTML and related files) and other resources that together constitute a web application.” All-in-one, that’s handy! Now, Tomcat is used to serve Java applications and one can add an application to it by what we call “dropping a war“. This means that Tomcat will unpack and start the application inside the war-file.

Our goal: we compile a war from the CloudStack source and then tell Tomcat to run it. Let’s me show you how we do this.

Installing Tomcat

In this blog I’ll show you how to do it on CentOS 7, as this is currently my favourite distribution. Tomcat runs on any OS, so with slight adjustments you can use another distribution as well.

yum install tomcat mkisofs

The package mkisofs is used by the Management Server. We’ll configure Tomcat later on.

The MySQL connector story

If you try dropping a CloudStack war in Tomcat, you will find that it doesn’t work because the MySQL driver cannot be found. My colleague Miguel Ferreira sent a Pull Request to add it but as it seems that wasn’t allowed due to licensing issues.

There’s a reason why the MySQL connector is not a dependency – it is Cat-X licensed, which means we may not depend on it in the default build.

Miguel then investigated alternatives and finally found that when you install the mysql-connector-java RPM package, and then put it at the front of the ClassPath, Tomcat will pick it up automatically. Details are in the Pull Request. Awesome work dude!

In short, this makes it work:

yum install -y -q tomcat mysql-connector-java
echo "CLASSPATH=\"/usr/share/java/mysql-connector-java.jar:${CLASSPATH}\"" >> /etc/sysconfig/tomcat

Building a CloudStack war-file

Time to build a CloudStack war-file. In order to do that, you need the CloudStack source and Maven + Java.

Clone the source:

git clone https://github.com/apache/cloudstack.git

Install these packages:

yum install maven tomcat mkisofs python-paramiko jakarta-commons-daemon-jsvc jsvc ws-commons-util genisoimage gcc python MySQL-python openssh-clients wget git python-ecdsa bzip2 python-setuptools mariadb-server mariadb python-devel nfs-utils setroubleshoot openssh-askpass java-1.8.0-openjdk-devel.x86_64 rpm-build

Set the Maven options like this:

export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n -Djava.net.preferIPv4Stack=true"

Set Tomcat options:

echo "JAVA_OPTS=\"-Djava.awt.headless=true -Dfile.encoding=UTF-8 -server -Xms1536m -Xmx3584m -XX:MaxPermSize=256M\"" >> ~tomcat/conf/tomcat.conf

Then go into the root of the source folder and run:

mvn clean install -P systemvm

If you have a fast computer, you may add -T 4 to compile with 4 treads. It’s faster, that’s all.

After a while you see:

[INFO] --------------------------------------------
[INFO] BUILD SUCCESS
[INFO] --------------------------------------------
[INFO] Total time: 06:09 min (Wall Clock)
[INFO] Finished at: 2016-02-29T11:21:50+01:00
[INFO] Final Memory: 109M/1571M
[INFO] --------------------------------------------

Notice this line:

[INFO] Building war: /data/git/cs1/cloudstack/dist/rpmbuild/BUILD/cloudstack-4.8.0/client/target/cloud-client-ui-4.8.0.war

So, the war is in client/target/ folder.

Copy the war-file to the Management server that will run Tomcat.

You can build the war-file on the management server itself, on another server, even on your laptop if you like. Just make sure you copy it to the management server (mine is named cs1).

scp client/target/cloud-client-ui-4.8.0.war [email protected]:/tmp

Dropping a war-file into Tomcat

The war-file needs to be added to Tomcat. You can use the UI manager for this and do it while Tomcat keeps running. To keep this demo easy, let’s stop Tomcat and assume it’s the only application it runs:

systemctl stop tomcat

Be sure to go to the Tomcat home folder:

cd ~tomcat/webapps/

Then copy the war-file you built, but rename it to ‘client.war‘. This is because the application is supposed to be called ‘client‘. This is why the url also has ‘/client‘.

cp /tmp/cloud-client-ui-4.8.0.war ~tomcat/webapps/client.war

Folder now looks like:

[root@cs1 webapps]# ls -al
total 247836
drwxrwxr-x. 3 root tomcat 4096 Feb 29 10:42 .
drwxr-xr-x. 3 root tomcat 4096 Oct 28 12:49 ..
-rw-r--r--. 1 root root 253766159 Feb 29 10:41 client.war

Starting Tomcat will automatically unpack the war and start it:

systemctl start tomcat

That’s it! Now, have a look at the folder:

[root@cs1 webapps]# ls -al
total 247836
drwxrwxr-x. 3 root tomcat 4096 Feb 29 10:42 .
drwxr-xr-x. 3 root tomcat 4096 Oct 28 12:49 ..
drwxr-xr-x. 11 tomcat tomcat 4096 Feb 29 10:42 client
-rw-r--r--. 1 root root 253766159 Feb 29 10:41 client.war

Management server log is here:

less ~tomcat/vmops.log

As you’ll see, CloudStack is started and the UI is available after a few minutes.

Missing files in war-file

If you want to deploy a new cloud, you’ll find that some stuff goes wrong. First of all, the database scripts are not in the war. You can place them in the Tomcat home dir.

To generate an archive of the files from the CloudStack source:

cd setup/db; tar -c -f ~/Downloads/db-scripts-${CLOUD_VERSION}.tar.gz -pPz db/; cd -

Copy this to the Tomcat homedir, and extract.

cd ~tomcat
tar zvxf db-scripts-${CLOUD_VERSION}.tar.gz
chown tomcat.tomcat ./db -R

The same counts for any additional scripts, like ‘cloudstack-setup-management‘ etc. They are not part of the war so you need to copy them should you want to use them.

Legacy /etc/cloudstack folder

If you’re looking for config files, they’re all inside the webapps folder. This means both a new location and that you need to backup/restore when dropping a new war-file.

You could make a symlink to the new location like this:

mkdir -p /etc/cloudstack
cd /etc/cloudstack
ln -s /var/lib/tomcat/webapps/client/WEB-INF/classes management 

Another option is to copy the files and add ‘/etc/cloudstack/management‘ to the CLASSPATH. The benefit is that you keep your config files out of the client folder so persistence is easier.

The user tomcat runs as

CloudStack wants it to run as user ‘cloud‘, don’t ask me why. For development it works fine as user tomcat, but for some stuff to work (like encryption) you better change it to user ‘cloud‘.

vim /etc/sysconfig/tomcat 
TOMCAT_USER="cloud"

Automation!

We obviously want automation in place for this to work without even thinking about it. You can use your favourite configuration management tool.

Flipping an attribute does the trick.

The work-flow we use looks like this:

– Build new war, then upload to some central place
– Flip a Chef attribute to indicate the new version should be installed
– Chef then takes over and:

– downloads the war-file
– downloads the db-scripts.tar.gz
– stops Tomcat
– backups settings, like db.properties and such
– removes old files
– puts new client.war in ~tomcat/webapps folder
– extracts db-scripts.tar.gz in Tomcat home folder
– starts Tomcat

Loadbalanced Management Servers

As you can see, there is some downtime involved when upgrading to a newer version. When you setup two or more CloudStack application servers with a loadbalancer in front of them, you can upgrade the management servers one-by-one without service interruption.

Conclusion

CloudStack clearly wasn’t designed for easy war-dropping into Tomcat, but as we’ve seen it can be done. Once you overcome a few hurdles, you’ll find it is actually much easier compared to upgrading RPMs all the time. Especially when you want to deploy the same version, with an extra bugfix this comes in handy.

It’d be nice to enhance CloudStack to fully support war drops. Even though users may want to use RPM packages we could simplify them by getting a war artifact (signed) from Maven Central or such, and then have the RPM package setup the things around it. We then would get the best of both worlds: Use RPMs if you want, be able to simply drop a war-file if you want to go faster.

For Schuberg Philis, this allows us to do continuous deployments to our production clouds. By making the numer of changes as smal as possible, we have many small deployments that are easy to test and easy to revert, should we have to.

Welcome to the new ‘continuous everything‘ world!