Remi Bergsma's blog

Inter-VPC connectivity made easy in Cosmic 5.3

Thu, 19 Jan 2017 20:48:50 +0100

It’s been one year after Cosmic started, forked off of the Apache CloudStack project. Cosmic 5.3 brings easy-to-use connectivity between VPCs. CloudStack users will recognise the Private Gateway functionality, which has been completely refactored in Cosmic 5.3. This blog shows what’s new.

The goal: connect VMs together that live in different VPCs
Basically, what you want is to setup a bunch of VPCs, create networks inside them where the VMs will live. Once you do that, there is no easy way to connect VMs from one VPC to another. Sure, you could make a VPN connection, and that is preferred in some cases. When you work in the same availability zone, and for example want Management VMs to access Acceptance or Production, there’s an easy alternative to get connectivity.

Workshop: Build Your Own Cloud In Record Time With Cosmic

Sun, 03 Jul 2016 10:31:17 +0200

At DevOpsDays Amsterdam 2016 I hosted a 3 hour hands-on workshop with my colleague Fred Neubauer. In these 3 hours we had 12 people build their own cloud based on Cosmic Cloud. It was great fun!

The slides are here:

View this presentation on SlideShare

Creating your own CloudStack release is easy!

Mon, 14 Mar 2016 21:21:56 +0100

I was the Release Manager for Apache CloudStack versions 4.6, 4.7 and 4.8 and during this time many people told me they thought it’s hard to build and package Apache CloudStack. The truth is, it’s not that hard once you know the steps to take. 😉

Next to that there’s the release pace. You may want to move in a different pace than the project is. There’ve been lots of discussions for example on a fast release cycle, and also on LTS-releases on the other hand. In this blog post I’ll show you how easy it is to create your own CloudStack release to satisfy the demands of your organisation.

How Tomcat war drops drive continuous deployments of CloudStack to production

Wed, 02 Mar 2016 11:25:42 +0100

Upgrading CloudStack can be tough at times. Some say therefore you should do as little upgrades as possible. My approach is to make it easy and make the number of changes as small as possible. The more upgrades, the better. The smallest unit is a single Pull Request with a single commit. That we should be able to deploy to production in an automated way with almost no effort. At Schuberg Philis, our current deploy record to our Mission Critical Clouds is five times a day. Here’s how we do it.

Video: Operating CloudStack the Easy Way (Automation!)

Thu, 12 Nov 2015 10:27:08 +0100

Watch my talk about how we do CloudStack Operations. It eas presented at the CloudStack Collaboration Conference in Dublin, on October 9th 2015.

The slides can be found in this post, and the code can be found on Github.

Watch this video on YouTube

This video is no longer available.

Video of my CCCEU keynote: The Mission Critical Cloud

Thu, 12 Nov 2015 10:21:54 +0100

Watch my keynote about the “The Mission Critical Cloud” at CloudStack Collaboration Conference in Dublin, on October 8th 2015.

The slides can be found in this post.

Watch this video on YouTube

This video is no longer available.

Slides from my talk: “Operating CloudStack: the easy way” at the CloudStack Collaboration Conference 2015 in Dublin

Fri, 09 Oct 2015 09:45:29 +0200

Here are the slides from my talk about CloudStack automation, at the CloudStack Collaboration Conference in Dublin today.

View this presentation on SlideShare

My keynote at the CloudStack Collaboration Conference in Dublin

Thu, 08 Oct 2015 09:15:36 +0200

Here are the slides from my keynote presentation at the CloudStack Collaboration Conference in Dublin today.

View this presentation on SlideShare

Tomorrow I’ll do another presentation and talk about automation cloud operations!

Rebooting a XenServer cluster while all VMs keep running

Sat, 11 Jul 2015 14:42:00 +0200

At Schuberg Philis, we manage quite a farm of XenServer clusters. As the number of clusters we operate goes up, so goes the time it takes to manage them all. And, let’s be honest: patching and upgrading isn’t the most exciting work. We needed more automation and I took the challenge 😉

The challenge
Patching itself is not a challenge, as we automated that already using Chef. The challenge comes when maintenance tasks require a hypervisor reboot in order to take effect. In such cases, some magic is needed to keep all VMs running, because we aim for zero down time due to maintenance. This blog describes how I automated rebooting full XenServer clusters while all VMs keep running.

Workshop: hands-on CloudStack at DevOpsDays Amsterdam

Wed, 24 Jun 2015 21:08:27 +0200

Today I hosted a CloudStack hands-on workshop at DevOpsDays Amsterdam, together with my colleague Fred Neubauer. It was really awesome to see all attendees building their own CloudStack cloud. And succeeding!

Here are the slides and instructions:

View this presentation on SlideShare

We proposed the workshop to the CloudStack Collaboration Conference Europe, in Dublin next October so we might do it again!

Fred in action!

That’s me in action!

Slides from my talk: “The Mission Critical Cloud” at the Amsterdam CloudStack meetup

Wed, 10 Jun 2015 18:30:07 +0200

Here are the slides from my presentation at the CloudStack meetup today in Amsterdam.

View this presentation on SlideShare

Get the code on Github.

Accepting pull requests: the easy way

Sun, 24 May 2015 21:13:47 +0200

Recently I became a committer in the Apache CloudStack project. Last week when I was working with Rohit Yadav from ShapeBlue he showed me how he had automated the process with a Git alias. I really like it so I’ll share it here as well.

First of all, pull requests are created on Github.com. Rohit created a git alias called simply ‘pr’. This is how it looks like (for the impatient: copy/paste the one-liner below). The command below is for easy reading, it will print a syntax error.

Making XenServer and CloudStack sing and dance together (again)

Sat, 23 May 2015 15:37:29 +0200

XenServer and Apache CloudStack work nicely together and form an easy-to-go couple for many companies. However, not many people know that recent releases changed the cooperation between the two drastically.

I’m not only talking about the shiny new Apache CloudStack 4.5 release, but the change was back ported to recent releases of 4.3.x and 4.4.x as well. This means that if you run XenServer and CloudStack, you need the review the information below carefully.

Making a XenServer unique after cloning from template

Thu, 07 May 2015 21:39:42 +0200

At work we’re creating a CloudStack development environment that can be created and destroyed as we go. One of the requirements is to quickly create a couple of hypervisors to run some tests. KVM is easy: they are just Linux boxes with some extra packages. XenServer is a different story.

Imagine you create a XenServer by installing it from ISO, then saving it as a template. Then create two instances from it called xen1 and xen2. If you try to add them to CloudStack, you’ll notice only the first one succeeds.

CloudStack Day Austin Video: Operating CloudStack – Sharing my Tool Box

Sun, 19 Apr 2015 20:10:22 +0200

Watch my talk “Operating CloudStack – Sharing my Tool Box” that I presented at CloudStack Day Austin, on April 16th 2015.

The talk was originally written for ApacheCon North America and I presented it there a few days earlier. It was great to be able to do the talk again and reach an even bigger audience!

The slides can be found in this post, and the code can be found on Github.

ApacheCon NA’15 slide deck: Operating CloudStack – Sharing my Tool Box

Mon, 13 Apr 2015 18:00:14 +0200

Here are the slides from my presentation at both ApacheCon and CloudStack Day in Austin, TX.

View this presentation on SlideShare

Get the code on Github.

Adding a KVM cluster with NSX networking to CloudStack

Sat, 04 Apr 2015 22:29:31 +0200

As KVM seems more and more interesting, at work we wanted to do a Proof-of-Concept. The KVM hypervisor cluster had to be controlled by CloudStack and also integrate with NSX (formerly known as Nicira).

NSX is owned by VMware these days and is one of the first Software Defined Networking solutions. At Schuberg Philis we use this since early 2012.

Choosing an OS
To me, the most interesting part of KVM is the fact you only need a very basic Linux box with some tooling and you have a nice modern hypervisor ready to rock. Since we’re using CloudStack to orchestrate everything, we do not need cluster features. In fact, this prevents the “two captain” problem that we sometimes encounter with XenServer and VMware ESX. We compared Ubuntu with CentOS/RHEL and both work fine. It all depends on your needs.

Connecting two Open vSwitches to create a L2 connection between sites

Thu, 26 Mar 2015 21:47:45 +0100

Recently I played with Open vSwitch and it’s awesome! Open vSwitch is a multilayer virtual switch and it brings a lot of flexibility in the way you can create interfaces and bridges in Linux. It’s also a Linux distribution independent way to configure these things. Switching in software!

To create a bridge, simply run:

ovs-vsctl add-br remibr0

You can also create another bridge on top of it, to handle a VLAN for example:

Pointing XenServer to a new Open vSwitch Manager (Nicira/NSX)

Tue, 17 Feb 2015 19:52:44 +0100

Our XenServer hypervisors use Nicira/NSX for Software Defined Networking (orchestrated by CloudStack).

We had to migrate from one controller to another and that could easily be done by changing the Open vSwitch configuration on the hypervisors, like this:

ovs-vsctl set-manager ssl:10.18.59.84:6632

It will then get a list of all nodes and use those to communicate.

Although this works, I found that when rebooting the hypervisor it would revert to the old setting. Also, when a pool master fail-over happened, Xapi ran a xe toolstack-restart and that caused the whole cluster to revert to the old setting. Oops.

CVE-2015-0235: Replacing the CloudStack systemvm template

Wed, 28 Jan 2015 23:59:58 +0100

Due to the Ghost bug aka CVE-2015-0235, we had to upgrade 500+ system vm’s. We’re running CloudStack 4.4.2. The version of the systemvm template it uses was 4.4.1 and so we created 4.4.2 and used that instead. It was quite some work to get it done so we thought it was worth sharing how we did it in this blog. I did this work together with my Schuberg Philis colleague Daan Hoogland.

1. Build new CloudStack RPM’s with MinVRVersion set to 4.4.2
Basically this was a single digit change in a single file (api/src/com/cloud/network/VirtualNetworkApplianceService.java).

HOWTO update a forked Git repo

Mon, 15 Dec 2014 13:28:14 +0100

When contributing to open source projects, it’s pretty common these days to fork the project on Github, add your contribution, and then send your work as a so-called “pull request” to the project for inclusion. It’s nice, clean and fast. I did this last week to contribute to Apache CloudStack. When I wanted to contribute again today, I had to figure out how to get my “forked” repo up-to-date before I could send a new contribution.

Be the automator!

Sun, 30 Nov 2014 19:11:15 +0100

Today I saw an awesome video of a presentation by Glenn O’Donnell.

In his presentation, Glenn states that it’s not about technology, it’s about services. Service design is modular with a logical structure. Approach it as a system and try to improve it as a whole, not just tiny pieces of it. To do that, you need systems engineers. Although most are lazy, or, as Glenn puts it: “Locally brilliant, globally stupid”.

CloudMonkey 5.3 on OSX Mavericks or OSX Yosemite

Fri, 24 Oct 2014 10:52:33 +0200

CloudMonkey is an easy way to interact with the CloudStack API without the need to do any code writing. Think of it like a way to “query” the CloudStack infrastructure. Very useful in day-to-day operations to get a “quick look” at things.

Version 5.3 is on its way, and recently I had a look at it. I used my MacBook Pro for this, and it came with some challenges to get it working. So, I’m writing down here how to get CloudMonkey 5.3 working on either MacOSX Mavericks (10.9) or the new Yosemite (10.10).

CCCNA’14 Video: Start using Configuration Management in 5 steps

Thu, 01 May 2014 08:07:10 +0200

Watch my talk “Start using Configuration Management in 5 steps”, at the CloudStack Collaboration Conference, Denver, Co, USA (April 9-11 2014).

Watch this video on YouTube

CCCNA’14 slide deck: Start using Configuration Management in 5 steps

Thu, 10 Apr 2014 23:15:07 +0200

Here are the slides from my presentation at the CloudStack Collaboration Conference in Denver:

View this presentation on SlideShare

Deep-dive into CloudStack again!

Sat, 05 Apr 2014 12:47:52 +0200

As you might have noticed, I switched jobs on Feb 1st and was quite busy after that. I joined Schuberg Philis, a Mission Critical Outsourcing company in The Netherlands. I met the folks on the CloudStack Collaboration Conference in Amsterdam, back in November. I have a very cool job, as I work with a bunch of very skilled colleagues on our Mission Critical Cloud, that is powered by Apache CloudStack.

Video recording of my talk at FOSDEM’14 in Brussels

Fri, 07 Feb 2014 19:23:59 +0100

Configuration management in an Enterprise Linux Team — How I automated myself out of my job

Watch this video on YouTube

Thanks to the FOSDEM video team for this video! It nicely integrates the slides and me talking! It’s just a pity the sound from the camera was used instead of the mic I was wearing. Therefore you hear a lot of noises from the people outside the room, on the hallway (there were >100 people that did not fit in the room anymore!). But anyway, enjoy the recording 🙂

FOSDEM’14 slide deck: Configuration management in an Enterprise Linux Team

Sat, 01 Feb 2014 12:22:00 +0100

Here are the slides of the talk I just gave in the Configuration Management devroom at FOSDEM’14:

View this presentation on SlideShare

Arrived @Brussels for my FOSDEM talk tomorrow!

Fri, 31 Jan 2014 22:52:31 +0100

How I ensure CFEngine is always running while also allowing an easy way to disable it

Tue, 14 Jan 2014 10:11:11 +0100

I’m currently finalizing the CFEngine 3 setup at my $current_work because by the end of the month I will start a new job. In a little over a year, I fully automated the Linux sysadmin team. From now on, only 2 sysadmins are needed to keep everything running. Since almost everything is automated using CFEngine 3, it’s very important that CFEngine is running at all times so it can keep an eye on the systems and thus prevent problems from happening.

Linux one-liner example to quickly automate a boring task

Sat, 04 Jan 2014 09:20:51 +0100

Recently a colleague came by and asked whether I could help him with finding out how much memory was already assigned to Oracle databases. It was fun to find out what was exactly he needed and to construct the one-liner bit by bit, as I detail below. When it seemed to work ok, I wrapped it in a small shell script to make it more flexible and reusable and deployed it to all Oracle database servers using CFEngine. You can find the final shell script on Github. I’d suggest you use the shell script instead of the one-liners below, as it is more accurate.

2013 in review: 733% increase in visitors and a challenging job done

Tue, 31 Dec 2013 11:17:43 +0100

The WordPress.com stats helper monkeys prepared a 2013 annual report:

Here’s an excerpt:

My blog was viewed about 220,000 times in 2013, which is a 733% increase over last year. On the last day of the year we even reached a quarter of a million total views! 🙂

Visitors came from 199 different countries this year. The busiest day of the year was March 19th with 1,640 views. The most popular post that day was Building an always available Linux terminal: Shell-In-A-Box on Raspberry Pi, that was featured on Linux Today.

Monitoring disk space with CFEngine

Mon, 04 Nov 2013 20:59:34 +0100

Now that I’m using CFEngine for some time, I’m exploring more and more possibilities. CFEngine is in fact a replacement of a big part of our Zenoss monitoring system. Since CFEngine does not only notice problems, but usually also fixes them, this makes perfect sense.

Recently I created a promise that monitors the disk space on our servers. Since we use Logstash to monitor our logs, all CFEngine needs to do is log a warning and our team will have a look. I will write about Logstash another time 😉

My tips for the Red Hat RHCE exam

Mon, 07 Oct 2013 23:19:59 +0200

Last week I attended the Red Hat RH300 course (fast track) in Amsterdam and did the RHCSA and RHCE exams on the final day. I passed both RHCSA (283/300 points) and RHCE (300/300 points). I had a great teacher because, apart from technical stuff, I also learned how to approach the exams.

The objectives for both RHCSA and RHCE are well documented on Red Hat’s site. You should start to make sure you know everything inside out. Practise, practise, practise. Learn to use the documentation that ships with RHEL, as this is the only help available: no internet access is provided during the exam. There are several books available that help prepare and Red Hat has very good courses as well, that I really recommend. I assume you should be able to study this all one way or the other.

Managing hundreds of Linux servers with CFEngine and Git: full control over configuration

Sat, 31 Aug 2013 22:59:02 +0200

Back in June, just before I went off for holiday, I attended a CFEngine training in Amsterdam. When I returned from holiday a few weeks later, me and my team started making plans to implement CFEngine in our environment. After two months of hard work, I’m proud to say we manage about 350 out of our 400 Linux servers with CFEngine!

The ride has been fun, although not always easy. In this post I’ll give a quick overview of our CFEngine implementation, where I found useful info, etc.

Improving data security: encrypting disks in Linux using LUKS and LVM

Wed, 31 Jul 2013 22:40:15 +0200

Data security is getting more and more important these days. Imaging you work as a sysadmin, on a laptop, and it gets lost. Of course you might lose your work (but you have a backup, right?). The real problem: your sensible data (SSH private key for example) is no longer under your control.

In this blog I explain how you can add an encrypted partition to Linux. As long as you also use a password protected screensaver, with a decent password (to protect a running, logged in laptop), no one can access your data. Even rebooting into single user mode (to by-pass the login screen) won’t help. No access to the encrypted disk without a working passphrase. Wiping your disk and reinstall is an option, but your data is not unveiled.

Today I started learning CFEngine 3

Thu, 13 Jun 2013 22:41:37 +0200

Me and two colleagues went to Amsterdam today, for a 1-day CFEngine-3 training. I’ve worked with configuration management before (Puppet), and my goal is to explore alternatives to be able to pick the right tool. Today’s quick overview training was a nice opportunity to get into CFEngine and meet some people behind the scenes.

Impression of CFEngine training

Diego Zamboni, author of the book Learning CFEngine 3, taught us what the concepts behind CFEngine are, how the language is build and how to get started. He demo’ed a lot of things and answered all of our questions. It was a very informative training that really inspired me.

Next step on the certification road: I am now LPIC-3 core certified!

Wed, 12 Jun 2013 19:12:48 +0200

Today I passed the senior level LPIC-301 core exam. I am now LPIC-3 certified!

The exam objectives are OpenLDAP and Capacity planning. Both of which I used in past projects, so I was already pretty familiar with them. In my current job I came across AD integration and Kerberos, and those were objectives as well.

In the past weeks I went through every detail of the objectives and learned some new stuff. My score on the exam was 690/800.

Playing with two-factor authentication in Linux using Google Authenticator

Sat, 08 Jun 2013 02:55:54 +0200

As discussed in a previous post, multi-factor authentication really makes things more secure. Let’s see how we can secure services like SSH and the Gnome desktop with multi-factor authentication.

The Google Authenticator project provides a PAM module than can be integrated with your Linux server or desktop. This PAM module is designed for for home and other small environments. There is no central management of keys, and the configuration is saved in each users home folder. I’ve successfully deployed this on my home server (a Raspberry Pi) and on my work laptop.

Suggestions for a strong password strategy

Fri, 07 Jun 2013 23:53:48 +0200

The recent article on Ars Technica, Anatomy of a hack: How crackers ransack passwords like “qeadzcwrsfxv1331” made me realize again how weak password authentication is. True, in this case the MD5 hashes were known, but still it’s only a single factor. Imagine what happens when you reuse a password on multiple sites, and one of them stores it as MD5 (or SHA-1, even plain-text) and gets comprimised.

Brute-force
What’s interesting in the above article, is that brute-force attacks only work up to 6 to 7 characters (depending on the hardware available) because of the so-called “exponential wall of brute-force cracking” caused by the exponentially increasing number of guesses each additional character creates. Although project Erebus v 2.5 already demonstrated brute-forcing any 8 character password in just 12 hours with a 8 GPU cluster. This, however, does not mean a 9+ character password is safe. In fact, passwords of 16 characters are easily cracked when using the right techniques.

Creating a multi hop SSH tunnel by chaining SSH commands and using a jump host

Tue, 28 May 2013 20:47:29 +0200

When it not possible to reach a server you want to SSH to directly, you can make use of SSH’s built in capability to chain multiple commands. Suppose you have a network setup like in the image below.

Network overview

Firewalls or ACL’s prevent direct access to the ‘web server’ in network #2. In between is a ‘jump host’ in network #1. A ‘jump host’ is a host you can SSH to, and from there reach the next hop. How to SSH to the web server?

How accurately can the Raspberry Pi keep time?

Sun, 12 May 2013 22:53:22 +0200

Recently I was wondering if a Raspberry Pi would be able to run an accurate clock and provide a stable NTP service.

A virtual machine can’t do it, because its system clock has a changing drift factor. The clock compensation technique built into NTP was never designed to handle anything but a static drift factor. This is unfortunate because NTP is a light-wight protocol, so using physical hardware for it may sound wasteful. The Raspberry Pi is physical hardware with a very small footprint. So, how accurately can the Raspberry Pi keep time?

Adding a hardware clock (RTC) to the Raspberry Pi

Wed, 08 May 2013 23:16:08 +0200

The Raspberry Pi does not have a hardware clock on board. Instead, to keep track of time during reboots and downtime the ‘fake-hwclock’ package is used. It contains scripts to save the kernel’s current clock periodically (including at shutdown) and restore it at boot so that the system clock keeps at least close to realtime. Combined with NTP this is a simple, cheap and fairly accurate setup for most use-cases.

One-liner to remove an incorrect entry in the SSH known_hosts file

Sat, 04 May 2013 19:41:16 +0200

When a SSH-key changed, this warning is displayed:

ssh -l username 172.16.12.34

Warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
aa:bb:cc:a2:b6:87:bd:43:f9:ff:<wbr />02:8e:a6:b8:29:42.
Please contact your system administrator.
Add correct host key in /home/remi/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/remi/.ssh/known_hosts:6

In case you know the host identification has changed, you can safely discard this warning.

Forwarding a local MySQL socket to a remote MySQL server

Fri, 03 May 2013 20:40:49 +0200

At times it can be very handy to be able to use a local MySQL socket, while the real MySQL server is in fact a remote one. I once used this technique to split a dozen LAMP servers into dedicated (clustered) web servers and MySQL servers. Without having all clients to update scripts, that is.

Today I ran into another use-case: while migrating an old 32bit MySQL Windows server to a modern Linux VM. We wanted to get some statistics from it. The famous ‘mysqltuner.pl‘ script is designed to run from localhost. Well, let’s fake MySQL is on localhost then 😉

HOWTO discover the ip address of a Raspberry Pi

Fri, 03 May 2013 19:20:40 +0200

When you boot your Raspberry Pi with Raspbian, it will get its ip address from a DHCP server. Most tutorials I’ve seen, ask you to login to a ‘router’ (that serves as DHCP server as well) to find the assigned ip address. When you haven’t access to the DHCP server, how do you find the Pi’s ip address?

You should know the MAC address of a Raspberry Pi always has a ‘b8:27:eb’ prefix. Using a Linux utility called ‘arp-scan‘ you can discover MAC addresses and their corresponding ip addresses. Arp-scan sends ARP packets to hosts on the local network and displays any responses that are received.

First, install arp-scan:

apt-get install arp-scan

Then locate all Raspberry Pi’s on the local network using this command:

arp-scan --interface=eth0 --localnet | grep b8:27:eb

You will need to be root, or arp-scan must be SUID root, in order to run arp-scan, because the functions that it uses to read and write packets require root privilege.

Result:

172.16.12.34 b8:27:eb:00:11:ab (Unknown)

Using the ip address, it is now possible to SSH into your Raspberry Pi:

ssh -l pi 172.16.12.34

Automatically adding and editing devices in Zenoss Core 4 using the API

Fri, 26 Apr 2013 20:54:54 +0200

Today I figured out how to automatically add new devices (in my case those are mostly virtual machines) to the Zenoss monitoring system. This used to be done by hand, but no more 🙂

To add a new device (for example a Linux server called server001), simply call:

curl -u apiUser:apiPass \
'http://zenoss-server:8080/zport/dmd/DeviceLoader? \
deviceName=server001&devicePath=/Server/Linux&\
loadDevice:method=1'

It’s wise to create a dedicated Zenoss user just for these API calls, but you may use any account that has sufficient permissions to perform the action you’re calling.

Building an always available Linux terminal: Shell-In-A-Box on Raspberry Pi

Fri, 15 Mar 2013 22:53:07 +0100

Recently I was looking for a way to SSH from a network that blocked my outgoing SSH connection. I’d be nice to have a way around firewalls and be able to access your private Linux terminal. To be able to debug a problem from an remote location, for example.

A collegue suggested a tool called ‘Shell In A Box‘. Shell In A Box implements a web server that can export arbitrary command line tools to a web based terminal emulator using just JavaScript and CSS without any additional browser plugins. This means: connecting your browser via HTTPS to your own hosted Shell In A Box web site, and access a Linux terminal from there.

Secure browsing on iOS (iPhone/iPad) using OpenVPN and the Raspberry Pi

Wed, 13 Mar 2013 23:08:21 +0100

A lot of people seem interested in safe browsing over untrusted wifi and using a Raspberry Pi with an OpenVPN server to accomplish that. In this blog post, I’ll continue the series and explain how to set up secure browsing on mobile devices based on iOS.

When I’m not at home I use my iPhone or iPad and connect to the internet using either 3G or public wifi. I want to configure my mobile device to setup an encrypted VPN tunnel and route all traffic through it, so I have a safe browsing experience. Fortunately, there now is an OpenVPN app available on iOS that allows you to do that! Please install this app using the App Store, and continue reading to set it all up.

Improving OpenVPN security by revoking unneeded certificates

Wed, 27 Feb 2013 15:30:57 +0100

OpenVPN is great, it allows for easy access in a secure way. But how do you keep it secure? I mean, what if someone leaves your company? Do you disable access to the OpenVPN server? You should! In this blog I’ll show you how to do it.

A feature called revoking exists in OpenVPN. Revoking a certificate means to invalidate a previously signed certificate so that it can no longer be used for authentication purposes. For this to work, we need to tell the OpenVPN server which certificates are no longer valid.

Passed the SUSE Certified Linux Engineer practicum (CLE)!

Fri, 22 Feb 2013 23:20:08 +0100

A few weeks ago I attended a full week of SUSE training, which I really enjoyed. There were lots of highly skilled geeks and the course had a nice solid pace. By the end of the week I did the CLE practicum exam (to become a Certified Linux Engineer). Today I heard I passed! 🙂

I’ve been working with SUSE Linux Enterprise Server since I started my new job back in December and the training really helped me.

Automating persistent iptables rules on Red Hat, SUSE and Debian

Tue, 12 Feb 2013 23:03:41 +0100

When you create firewall rules with iptables on Linux, you want to make them persistent over reboot, because they are not by default. Different Linux distributions have different methods of achieving this, although the basics are similar. I’ve been working with Debian, Red Hat Enterprise Linux and SUSE Linux Enterprise Server and in this blog I’ll describe how to configure each of them to save your iptables rules across reboots.

Secure browsing via untrusted wifi networks using OpenVPN and the Raspberry Pi

Sun, 27 Jan 2013 23:02:34 +0100

The Raspberry Pi can easily be setup as an OpenVPN server. One common feature is to access servers or services on the remote network. Another use case is to provide a secure connection when you’re not at home.

You can do this by sending all traffic over the VPN connection, instead of only traffic for the remote servers. The idea is to connect your laptop to your OpenVPN server (this is encrypted by default) and access the internet from there. Whatever you send over the wifi connection, is encrypted by the VPN and thus safe. In the comments on the previous post there was someone trying to set this up. To help him solve the issue, I tested this myself, and decided to write a post showing how to do it because I really like the idea.

Using key-based ssh authentication with ssh-agent and GNU screen

Wed, 23 Jan 2013 12:25:26 +0100

I’m using SSH’s key-based authentication to login to my servers using a SSH key pair. Of course my private key is protected with a password of its own. This has the consequence that even with key-based authentication, you still need to type a password: your private key passphrase. And you need to type it on every single connect. Although this is far more secure that having no passphrase on your private key, typing it so many times is a bit over done.

Passed LPIC-2 today!

Wed, 16 Jan 2013 19:42:15 +0100

About 2 weeks ago I wrote in a previous post about LPIC, when I had just passed the LPIC-1 exams. In the mean time I studied the objectives, read some man-pages and experimented with the things I wasn’t 100% familier with.

And today I completed LPIC-2! 🙂 There were two exams required to pass. I scored 740/800 on the 201 exam, and 730/800 on the 202 exam.

HOWTO connect to hosts on a remote network using OpenVPN and some routing

Sun, 13 Jan 2013 22:06:26 +0100

OpenVPN is a very powerful tool to connect to a remote network in a secure and easy way. But while it’s quite easy to set it up (see my post about OpenVPN on Raspberry Pi), getting all the pieces together takes some more network understanding and configuration. I created an image to show what’s going on when a OpenVPN client connects to the server.

Transforming a website into a static one that works off-line using wget

Sun, 13 Jan 2013 15:50:36 +0100

Someone asked me if it were possible to download a web site and make it available offline. To some extend, this can be done. Interactive forms will not work (searching, ordering, etc), but you can use ‘wget‘ to transform a website into a static version.

It goes like this:

wget \
 --recursive \
 --no-clobber \
 --page-requisites \
 --html-extension \
 --convert-links \
 --restrict-file-names=windows \
 --domains example.org \
 --no-parent \
 --wait=1 \
 --limit-rate=500K \
 example.org/

Let me explain:
The ‘–recursive’ option downloads the entire web site and ‘–domains’ tells wget not to follow links outside example.org. Otherwise you will download far too many pages. ‘–page-requisites’ makes sure we’ll get all the elements that compose the page (images, CSS, etc), ‘–html-extension’ saves files with the .html extension so they will work on a stand-alone pc, ‘–convert-links’ converts links so they’ll work off-line and ‘–no-clobber’ prevents any existing files to be overwritten

Building an economical OpenVPN server using the Raspberry Pi

Sat, 05 Jan 2013 23:53:31 +0100

The Raspberry Pi is a $35 credit-card sized computer, with an ARM-based CPU. It uses very little power (only 3 Watt), so it’s ideal for a server that’s always-on. I was thinking what’d be a nice task for my Raspberry Pi and came up with an OpenVPN server. This enables me to connect to my home from anywhere, for example to access some files or to access the internet from there.

2012 in review: my blog was read by people from 165 different countries

Mon, 31 Dec 2012 14:34:21 +0100

The WordPress.com stats helper monkeys surprised me today with their 2012 annual report.

Here’s an excerpt:

My blog had 33,000 views in 2012. The busiest day of the year was September 4th with 644 views. The most popular post that day was How I replaced the CloudStack Virtual Router with my own physical Linux router that was featured on Linux Today.

Visitors came from 165 different countries.. wow! Most came from The United States, followed by The Netherlands and Germany. A lot of the visitors were searching on Google, some on Twitter, and found my blog that way.

First results in getting my Linux knowledge certified: LPIC-1 and CLA

Fri, 28 Dec 2012 23:04:58 +0100

After passing my Bachelor of Computer Science I never took the time to take any specific Linux exam. But I’ve been working with Linux for about 14 years, so I know a lot about it. Now that I started my new job this month, I thought it was also time to get Linux certified.

I’ve been given the opportunity to get some training from SUSE (which is the distribution we use at work), and I’m really happy about that. Yes, I do have a nice employer 🙂

Using Exchange services on a Linux Desktop with DavMail

Tue, 11 Dec 2012 22:43:39 +0100

Since I started my new job last week, I no longer work on a Mac desktop. I now use Ubuntu instead. Our company has an Exchange server and uses Active Directory for authentication. Using a Linux desktop in a Windows environment can be a bit challenging. While you can use Thunderbird to use imap, features like calandar and global address book are not available. Of course there’s Terminal Services using RDP but this is not ideal. My new colleagues pointed me to a little tool called DavMail.

Make the mouse wheel in Ubuntu behave like Mac OSX

Fri, 30 Nov 2012 15:12:30 +0100

I’m used to my Mac where the direction of scrolling was changed to a more ‘natural’ way starting with OSX Lion. It’s the same way scrolling works on mobile devices. When working on Ubuntu, I want to have the same behaviour because I’m used to it.

To do so you need to edit (or create) the .Xmodmap file in your homedir.

This is the default:

cat ~/.Xmodmap
pointer = 1 2 3 4 5

I changed it to:

Fixing an out-of-sync or corrupt MySQL slave

Tue, 20 Nov 2012 21:28:13 +0100

Sometimes a MySQL slave may get corrupted, or its data may otherwise be unreliable. Usually I clone the data from a slave that is still ok and fix it from there. However, today I run into an issue that made me doubt on the data of any slave. To be absolutely sure the data is consistent on both master and slave, I decided to deploy a new slave with a clone of the master and then redeploy the other slaves from the newly created slave like I normally do with a script.

Replacing both DRBD nodes while keeping the storage online

Mon, 19 Nov 2012 22:34:04 +0100

DRBD (Distributed Redundant Block Device) is an open source storage solution that is best compared with a RAID-1 (mirror) between two servers. I’ve implemented this for both our cloud storage as our cloud management servers.

We’re in the process of replacing both cloud storage nodes (that is: everything except the disks and its raid array) and of course no downtime is allowed. Although DRBD is made for redundancy (one node can be offline without impact), completely replacing a node is a bit tricky.

Securely connect an Advanced (private) CloudStack network by using a VLAN aware switch and a basic VPN device

Sun, 18 Nov 2012 22:45:49 +0100

I’ve been working a lot with CloudStack Advanced networking and find it very flexible. Recently, I had another opportunity to test its flexibility when a customer called: “We want VM’s in your CloudStack cloud, but these VM’s are only allowed be reachable from our office, and not from the public internet”. Firewalling? No, they required us to use their VPN solution.

Is CloudStack flexible enough for this to work? Yes, it is. In this blog I’ll tell you how we did it. And it doesn’t even matter what VPN device you use. This will work regardless of brand and features, as lang as it supports a public ip-address to connect over the internet to another VPN device, and has a private network behind it. All VPN devices I know of support these basic features.

Migrating an ip-address to another server: clear the arp cache of your neighbors

Thu, 15 Nov 2012 20:31:40 +0100

When migrating an ip-address to another server, you will notice it will take anywhere between 1 and 15 minutes for the ip-address to work on the new server. This is caused by the arp cache of the switch or gateway on the network. But don’t worry: you don’t just have to wait for it to expire.

Why it happens
ARP (Address Resolution Protocol) provides a translation between ip-addresses and mac-addresses. Since the new server has another mac-address and the old one stays in the cache for some time, connections will not yet work. The cache usually only exists for some minutes and prevents asking for the mac-address of a certain ip-address over and over again.

Some handy Git commands I often use

Sat, 10 Nov 2012 23:15:13 +0100

Push a specific commit (actually: everything up and including this commit):

git push origin dc97ad23ab79a2538d1370733aec984fc0dd83e1:master

Push everything exept the last commit:

git push origin HEAD~:master

The same, now the last two commits:

git push origin HEAD~2:master

Reorder commits, aka rebasing:

git rebase -i origin

Pulling commits from repo to local

git pull --rebase

When a conflict occurs, solve it, then continue:

git rebase --continue

Put local changes apart (shash them)

git stash save stashname

Show all stashes

What a battery means for the performance of our raid array

Fri, 02 Nov 2012 22:40:13 +0100

A few days ago we installed a ‘Battery Backup Unit’ in our secondary storage server. This allows us to turn on the ‘Write Back Cache’. The performance impact was impressive..

Enabling the Write Back Cache means writes are committed to the raid controler’s cache (which is much faster) so you don’t have to wait for the data to be written to disk. Normally, this is a risky operation because when the power goes down unexpectedly the data in the raid controller’s cache is lost. Thanks to the battery the raid controller can finish all of its writes to the disk even there is no more power.

Block web traffic based on ip-address and display an error message

Tue, 23 Oct 2012 16:48:29 +0200

Sometimes it is necessary to block access from a certain ip-address. This can be done easily using route:

route add -host 1.2.3.4/32 reject

While this works, it does not provide the best user experience because from 1.2.3.4 the website now seems down, while it isn’t. A better way is to display an error message instead of the website requested.

I’m using load balancing to distribute the load to different web servers. The software in use is Keepalived. To block a given ip-address, I have the firewall tag it and then make Keepalived forward it to another web server instead. It goes like this:

Backup MySQL and automatically compress the backup

Tue, 23 Oct 2012 16:02:45 +0200

Here are three ways to backup your MySQL database(s) and compress the backup on the fly:

1. Dump database ‘dbname’ into a compressed file

mysqldump \
-u user \
-ppassword \
--host=mysql-server \
dbname | \
gzip > dbname_backup.sql.gz

To dump multiple databases, you can either write a ‘for loop’ on a ‘SHOW DATABASES’ query result or use the next command to backup to a single file.

2. Dump all databases into a compressed file

Labelling cables the right and easy way using the Brady BMP21

Wed, 10 Oct 2012 15:20:53 +0200

When building new computer infrastructure, I find it important to label and document everything properly. This will help you in an emergency, especially if you are not on site and need to instruct someone on the phone. Apart from that, todays infrastructure is quite complex and one cannot remember everything by head. This blog post describes what we used when building our Cloud infrastructure.

First of all, use a separate cable color for each type of connection. We used, for example, different colors for ‘storage’, ‘management’, ‘serial’ and ‘public’ traffic types. This gives a quick overview of the different cables that are in use. Make sure colors are unique, so don’t use black if you’re using black power cables.

Save your Linux shell commands and their output for later review

Mon, 08 Oct 2012 22:46:07 +0200

Changes to production systems should be tested on a development system and then be deployed using configuration management (such as Puppet), if you ask me. Sometimes I first run commands by hand (on a test system) to find out the right ones, the right order, etc. In those cases I find it useful to automatically document what I type and be able to ‘replay’ it later on.

Why? Because when I write the Puppet configuration needed to deploy the change, I want to be sure all manual commands I ran make it to the Puppet manifest. Being able to replay what I did allows me to do so with ease.

Easy to fix but dangerous vulnerability in CloudStack API discovered

Mon, 08 Oct 2012 21:38:13 +0200

It just came to my attention that a vulnerability in Apache CloudStack was discovered, as John Kinsella writes in his post to the Apache CloudStack dev-mailinglist.

A malicious user could, for example, delete all VMs in the system. Addressing this issue is especially important for anybody using CloudStack in a public environment.

The vulnerability report has an easy work-around that I will mention here as well:


mysql -p -u cloud -h mgt-server-ip
update cloud.user set password=RAND() where id=1;
\q

Hugo Trippaers of Schuberg Philis discovered this issue. Thanks for sharing!

One-liner: restore compressed MySQL backup

Mon, 08 Oct 2012 20:35:56 +0200

One-liners are always fun. Linux has pipes, streams and redirects which you can combine to do many things in just one line. Today I had to restore a MySQL backup that was gzipped. Instead of typing:


gunzip db-backup.sql.gz
mysql -p -h db.example.com dbname < db-backup.sql

You can type this instead:


gunzip < db-backup.sql.gz | mysql -p -h db.example.com dbname

The .sql.gz file is input to gunzip, the resulting (unzipped) data is piped to mysql and is used as input there.

Fixing MySQL replication after a faulty query

Fri, 28 Sep 2012 19:23:24 +0200

Today I run into an issue with MySQL replication that prevented updates from the master to appear on the slaves. When checking the slave status, MySQL reports:

Slave_IO_State: Waiting for master to send event
Slave_IO_Running: Yes
Slave_SQL_Running: No
Last_Error: Query caused different errors on master and slave.

Apparently, an error occurred on the master (like a query that was wrong) and that failed on the slave as well. But since the error messages differ (a bit) the replication was stuck on this query. Manual intervention is required to tell MySQL what to do next.

Import your Bank Account transactions into the Account Tracker iPad app with ease

Tue, 18 Sep 2012 19:40:59 +0200

To track my spendings, I started using the Account Tracker iPad app by Graham Haley. It’s a very solid and well written app that I’ve been using for some weeks now. I really like it, especially because it has a clearly designed gui and rich feature set. Another big plus is that the developer responds very quickly.

But, entering all transactions by hand for me was a no-go. I just don’t want to do that because it’s a lot of work and we might also forget to enter some. Furthermore, almost all transactions we do are done electronically. That’s when I started to think about importing transactions from our Bank Account into the iPad app. Fortunately the app has a feature that let’s you import a csv file. Our bank’s website also has an export feature so that should be doable!

Sed inline editing different on Mac OSX?

Tue, 18 Sep 2012 10:48:38 +0200

Sed is very powerful, I use it a lot on Linux servers I manage. Today I was working on a local git repository on Mac OSX Mountain Lion when I run into some trouble.

Usually, to replace text in a file with new text I run:

sed -i 's/Find this text/Replace with this/' file_to_replace_in.txt

While this works on Linux, it does not on Mac OSX:

sed: -i may not be used with stdin

The manpage on OSX says:

Building a redundant pair of Linux storage servers using DRBD and Heartbeat

Sun, 09 Sep 2012 22:20:16 +0200

I’ve been building redundant storage solutions for years. At first, I used it for our webcluster storage. Nowadays it’s the base of our CloudStack Cloud-storage. If you ask me, the best way to create a redundant pair of Linux storage servers using Open Source software, is to use DRBD. Over the years it has proven to be rock solid to me.

DRBD is a Distributed Replicated Block Device. You can think of DRBD as RAID-1 between two servers. Data is mirrored from the primary to the secondary server. When the primary fails, the secondary takes over and all services remain online. DRBD provides tools for failover but it does not handled the actual failover. Cluster management software like Heartbeat and PaceMaker are made for this.

How I replaced the CloudStack Virtual Router with my own physical Linux router

Thu, 30 Aug 2012 23:52:32 +0200

Back in March I wrote a blog on how to create a network without a Virtual Router. I received a lot of questions about it. It’s also a question that pops up now and then on the CloudStack forums. In the meanwhile I’ve worked hard to implement this setup at work. In this blog I’ll describe the concept of working with a CloudStack setup that has no Virtual Router.

How do you think I can improve my blog?

Fri, 17 Aug 2012 20:31:25 +0200

Now that I have some time off, I enjoy spending some of it on my blog. I played with some WordPress settings and redesigned the right column. My most important goal here is to make sharing easy.

Why? Because I benefit a lot from what others post and want to give back a bit. That’s why I write my blogs. If you find them useful, it’s now easy to share them on Twitter, Facebook, Linked-In and others. The more people benefit, the better, don’t you think?

Enabling Zimbra auto responder for multiple e-mail addresses

Thu, 16 Aug 2012 13:11:58 +0200

After all the hard work in the past months it’s now time for some weeks off 🙂 One last thing to solve was the Zimbra auto responder: I had enabled it through the web interface but while testing it, I found it only replied mails sent to my main e-mail address. There is no way to fix this in the web interface, so I had a look at the command line options and found a way to add extra e-mail addresses. Ssh to the machine, become user ‘zimbra’ and run this command:

HOWTO remove null characters from a file

Wed, 15 Aug 2012 21:12:57 +0200

Sometimes files may be filled up with null characters that look like ^@ when you open them in a text editor. This may happen when a disk becomes full, or when you rename a logfile while an application is still writing to it.

I ran into this problem today, and I fixed it using a command called ‘tr’. This is a utility capable of translating or deleting characters from standard input/output. It means you can use it to ‘pipe’ input to it, and send the output to a new file. For example:

Stopping spam sent by botnets effectively: combining Passive OS Fingerprinting, Greylisting and Spamassassin

Tue, 14 Aug 2012 17:02:00 +0200

At work, it is of my tasks to make sure we keep the mailboxes of our clients free of spam. Some weeks ago the number of spam mails went up massively and we worked hard to update the filters to keep unwanted mails out. In this blog I’ll describe a few of the things we did.

Using the famous SpamAssassin tool it is possible to score e-mails. One can score on the contents of subject, body, headers etcetera. A lot of good rules are already supplied and it’s possible to write your own. When a new spam run comes in, we used to create new rules for the spam mails that slipped trough. That works, and afterwards the mails are tagged as spam.

Sorting Apache2 logfiles from multiple webservers by date so AWStats processes them correctly

Sat, 11 Aug 2012 10:54:07 +0200

I had an interesting problem lately regarding AWStats. Due to some delay, the log files weren’t processed in the right order and then AWStats ignored all old logs. This resulted in some days being blank in the stats and of course this is not something we want. Since we also have multiple web servers in our cluster, things started to get a bit complicated.

The log files from each of the web servers were concatenated and then split to a separate log file for each virtual host using the Apache2 split-logfile script.

Restoring iPhone after repair very easy with iCloud

Sat, 04 Aug 2012 21:02:18 +0200

My mother’s iPhone had issues and was replaced by a new one. That did Apple handle very well 🙂 The down side of a new iPhone is, of course, that all settings and photo’s were lost. Fortunately, I was smart enough to enable iCloud backups last year when I setup her iPhone.

So, when the new iPhone came in today, all I did was restore the iCloud backup in a few swipes and clicks. An hour or so later (due to downloading) everything was as before. Very cool!

OpenVPN on Mountain Lion: Tunnelblick alternative Viscosity to the rescue?

Fri, 03 Aug 2012 20:21:21 +0200

Last week I wrote a small blog about OpenVPN on OSX Mountain Lion using Tunnelblick. I managed to get things to work using the latest Tunnelblick beta. But after a week working with it, I’m not too happy how it works right now. The two issues I have are:

1. DNS servers do not always get set properly, it feels unstable to me. I’ve seen /etc/resolv.conf with the right content, but still the old DNS servers were used by Mountain Lion. This is annoying: while connected to the vpn, hosts behind it do not resolve because the DNS server advertised by the OpenVPN server is not being used. This happened a few times last week, although most of the time it works ok. What goes wrong more often is the opposite: when disconnecting the connection, the OpenVPN advertised DNS server is still being used; but since we’re disconnected, it doesn’t resolve anymore and so nothing works. I’ve then to manually restore the right DNS servers. No fun, but I accepted it for now. I thought there’d be an update soon enough that’d solve this.

Overriding default keyboard shortcuts in Mac OSX (Mountain Lion)

Sun, 29 Jul 2012 12:16:21 +0200

I’ve upgraded my 3 year old MBP with a fast SSD. I decided to install from scratch instead of migrating or cloning the old disk. The good thing is that everything is now fast and clean. The downside is that all customisations I did over the past years are also gone.

One thing I customised was shortcuts. I forgot how easy this can be done in OSX, so I’ll describe it below so others can benefit as well.

Fixing OpenVPN/Tunnelblick DNS issues on OSX Mountain Lion

Sat, 28 Jul 2012 12:59:24 +0200

I just upgraded to OSX Mountain Lion and had some trouble this morning getting my OpenVPN connection (using Tunnelblick) to work. It would connect, but the DNS resolving was unstable. I managed to get it to work on the latest stable by changing the DNS-server setting to ‘set nameserver 3.0b10’ but then internet traffic would not work. The best solution for me at the moment is to upgrade to Beta release ~~3.3beta16~~ 3.3beta18.

Removing old equipment from datacenter

Sat, 21 Jul 2012 22:57:46 +0200

We’ve virtualized many servers already this year. Last month we moved the MD equipment that still needs to be virtualized. Today we removed the old MN servers that are no longer needed due to our new private cloud. All our main equipment is now together in the DC-2 datacenter in Amsterdam!

Moving a MySQL slave to a new replication master

Tue, 10 Jul 2012 22:28:09 +0200

I’m upgrading our MySQL master/slave setup and am moving it to new (virtual) hardware in our cloud environment. One of the things I did last night was moving the MySQL slaves to a new master that I had prepared in the new environment. This post describes how I connected the slaves to their new master in the cloud.

First, you’ll need to make sure the new master has the same data as the old one.
1. Make sure no more updates occur on the old master
2. Create a sql dump of the master using mysqldump
3. Import that dump into the new master using mysql cmd line tool

Setting locales correctly on Mac OSX Terminal application

Tue, 10 Jul 2012 21:51:58 +0200

I’ve had some trouble when using the Mac OSX Terminal app for some time now. Until today, it just gave me some annoying warnings from time to time. Like when installing an application with apt-get in Debian:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = “UTF-8”,
LANG = “en_US.UTF-8”
are supported and installed on your system.
perl: warning: Falling back to the standard locale (“C”).
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

Video impression of MD datacenter move 2012

Sun, 01 Jul 2012 22:46:52 +0200

After many weeks of preparation we moved MD’s equipment from the EUNetworks datacenter in Amsterdam, to Gyrocenter DC-2. This is also our new datacenter where we’ve built a CloudStack cloud.

Most servers were virtualized before this migration and only had to be removed during this operation. Others will be virtualized in the coming months. Fortunately most services had only some downtime during the night.

We also moved MD’s backup location (Redbus) and integrated it with our equipment in Global Switch.

High cpu load on CloudStack management servers after leap second 30/6/2012@23:59:60 UTC

Sun, 01 Jul 2012 07:46:53 +0200

After the leap second insert last night, my CloudStack 3.0 servers (or its Java processes actually) started to use a lot of CPU. Here’s how to fix it (sets the date):

date ; date -s "`date -u`" ; date

Just run this on your management- or compute node. ~~On my CloudStack system it only occurred on the management servers.~~ Confirmed: it occurred both on management- and compute nodes on our CloudStack system. No restart required afterwards . The load will drop immediately.

Meet our shiny new CloudStack Cloud

Thu, 28 Jun 2012 14:24:56 +0200

This is what our Cloud looks like in the CloudStack Dashboard. Pretty powerful 🙂

These are the boxes the hardware was in.

This image is taken in our lab while testing CloudStack.

Very handy: a tray for spares in your rack!

This is the front: 6 compute nodes, 2 management servers and 2 Linux routers that manage all traffic. We also have 2 big storage servers that you cannot see on this image.

Upgrading Keepalived in Debian Squeeze

Sat, 16 Jun 2012 23:26:04 +0200

I’m running Keepalived on our loadbalancers for many years and I’m really happy with it. Today I run into an issue that took me some time to solve. I thought I’d share it 🙂

In my current setup I have a pair of Debian Squeeze boxes running version 1.1.20. Since I’m rolling out ipv6 at the moment, I need to upgrade to 1.2.2. Fortunately, Debian provides this version in Squeeze-Backports.

So, I decided to upgrade the Backup loadbalancer first (lb-1). It was a simple ‘apt-get’ procedure to get it installed. But soon these errors popped up in my syslog:

High Availability with automatic failover for CloudStack management servers

Wed, 02 May 2012 14:14:11 +0200

I’m using CloudStack for some months now and our cloud is close to going live. It’s an awesome piece of software that is just rock solid :-). One thing I couldn’t really find is how to create high available management servers with automatic failover. I’ve come up with a solution that I’ll share in this blog post.

From the CloudStack manual:

The CloudStack Management Server should be deployed in a multi-node configuration such that it is not susceptible to individual server failures. (…) MySQL may be configured to use replication to provide for a manual failover in the event of database loss.

RRDtool: moving data between 32bit and 64bit architectures

Mon, 30 Apr 2012 13:15:49 +0200

When you mix architectures (or are moving from 32bit to 64bit like me) and use RRDtool for statistics, you might run into this problem:

ERROR: This RRD was created on another architecture

You can solve this, to dump the RRD file on the server that created it to XML, like this:

rrdtool dump stats.rrd > stats.xml

Then transfer it to the new server, and convert it back to RRD:

rrdtool restore -f stats.xml stats.rrd

The other server can now read the file. You can test it with the info command:

Using GNU Screen to manage terminal sessions

Sun, 29 Apr 2012 11:55:05 +0200

Whether you’re using Mac OSX, Windows or Linux, we’re all using a so-called “window manager”. Most are graphical user interfaces; and that’s a good thing 🙂 But as a sysadmin I need to manage many servers. Servers without a graphical user interface. So, how to handle that?

One could just ssh into a server when you need to do work on a given server. Depending on the terminal program you use, you might be able to have multiple sessions at the same time, preferably in tabs. It looks like this when 3 tabs are open:

HOWTO enable RPMForge repository in CentOS

Wed, 18 Apr 2012 21:56:56 +0200

Since I’m working with CloudStack I’m also working with CentOS. As I’ve been working with Debian for over 10 years, it sometimes takes some extra time to get things done. Like when you want to install a package that is not in the default repo’s.

This is how to enable the RPMForge repository:

Start to import the key:

rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt

Then download the RPMForge release file:

wget http://apt.sw.be/redhat/el6/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm

Then all you need to do is install this rpm:

HOWTO create php5-PECL Debian packages

Thu, 05 Apr 2012 20:01:07 +0200

The PHP Extension Community Library (PECL) has many great extensions for php5. Unfortunately, most are not packaged for Debian. Of course it is pretty easy to install it by hand. For example the ‘pecl_http’ extension:

pecl install pecl_http
echo "extension=http.so" > /etc/php5/conf.d/http.ini

For this to work you need some -dev packages to be installed. php5-dev and build-essential are the minimum required.

I have two problems with this method:
1. Too many -dev packages will be installed on production servers. I want to have as less packages installed as possible;
2. The software is not easily installable and upgradable;

The power of coding your infrastructure – CloudStack and Puppet in action!

Sun, 25 Mar 2012 23:37:47 +0200

Lately I’ve been coding to automate the deployment of Virtual Machines that will run in our cloud. It’s inspired by blogs and code from Jason Hancock but implemented in a different way to meet our specific needs. Since CloudStack has a very flexible API, you can make it do anything you want. It works pretty cool so far.

From a command line script we can deploy a new VM with a certain “role”. This role has a name, like “web server” or “test-vm-db” and contains the OS template name, the datacenter name, the networks to connect to, the hardware offering (CPU/Memory/Disk) and the account and domain to use. These are all CloudStack specific settings. When calling the deployVirtualMachine() API command, CloudStack will create a VM with exactly these specs. We use a simple php cli script to add a new server, like this:

DevOps: The new way of doing sysadmin and development

Sun, 25 Mar 2012 22:38:53 +0200

Today while I was running I realized I’ve been doing a lot of coding lately. Coding our new infrastructure to be exact. I remembered Kris Buytaert’s talk about DevOps back in February when I was in Antwerp. One of the key statements is that there are ‘sysadmin coders’ and not ‘sysadmins’ and ‘coders’. The only way to achieve great results, is when these two work together and communicate with each other. The idea’s behind this are called DevOps. IT Operations starts using code to manage configurations and infrastructure instead of doing it by hand over and over again. Thanks to CloudStack and Puppet this is now possible. Ideally, you would not have two groups, but one. Stephen Nelson-Smith from jedi.be describes it like this:

HOWTO automatically reboot a Linux system after a kernel panic

Wed, 21 Mar 2012 21:18:02 +0100

Remote Linux systems usually have to be available all the time. Although Linux is rock solid and stable, an occasional crash can occur. For example when there’s a problem with hardware or software. The last thing you want, is that a Linux server got a kernel panic and then waits forever for someone to reboot it.

Of course there’s monitoring, and there are APC’s too, which can reboot the server. This is like pulling the plug. It usually takes some time for monitoring to notify a sysadmin and then for the sysadmin to reboot the server. And, when using IPMI devices (especially the older ones that share a network connection with the server), a kernel panic could make them unavailable, too. I’ve had that on many occasions. Then you end up driving to the data center or call someone to reset the server. I hate that 😉

Great run, great weather!

Wed, 21 Mar 2012 12:55:54 +0100

Today is a very nice day with a lot of sun shine. This morning I decided to go for a run, to get rid of the stress and to enjoy being outside. Just running for some time makes me calm and relax again. I love that feeling!

I haven’t been running for almost a year. Today I took my Garmin Forerunner 305 with me. The nice thing is that the device has a GPS chip and can track many things, like distance, pace, heartbeat, calories etc. When I got home, I uploaded the run data to the MapMyRun website. It produces these nice images:

HOWTO enable color for PHP and Bash scripts in Nano

Thu, 15 Mar 2012 22:48:03 +0100

When working in shells, it’s very useful to have syntax highlighting. In vim it’s just ‘:sy on’ and you’re done. For those of you using GNU Nano, I’ll show you how to enable syntax highlighting as well.

To start, we’ll need some files which tell nano what to display in what color. Here is an example of such a file for PHP:

nano /usr/share/nano/php.nanorc
syntax “php” “.php|.inc$”
color white start="<?(php)?" end="?>"
color magenta start="<[^?]" end="[^?]>"
color magenta “$[a-zA-Z_0-9]”
color brightblue “->[a-zA-Z_0-9]”
color cyan “([)|(])”
color brightyellow “(var|class|function|echo|case|break|default|exit|switch|if|else|elseif|@|while|return|public|private|proteted|static)\s”
color brightyellow “&lt;(try|throw|catch|operator|new)&gt;”
color white “=”
color green “[,{}()]”
color green “==”
color brightgreen “(’[^’]’)|("[^”]")"
color yellow start=""
color yellow start="/*" end="*/"
color yellow start="#" end="$"
color yellow “//.*"

Tweak the OSX Terminal to display colors

Thu, 15 Mar 2012 22:17:06 +0100

Today I came across a nice post by Tod Werth, who created a nice theme for the OSX Terminal program called IR_Black.

All you have to do is download his schema and open in it Terminal. Then tweak the colors a bit to fully meet our needs.

Last step: enable colors in your profile:

vim ~/.bash_profile

Add this line:

export CLICOLOR=1;

When you open a new window and enter a simple ‘ls’ command, it looks like this:

HOWTO create a network in CloudStack without a Virtual Router

Sat, 10 Mar 2012 21:49:20 +0100

Networking in CloudStack 3.0 is awesome; the Virtual Router provides many cool features like LoadBalancing, PortForwarding, (s)NAT, DHCP, VPN and so on. When a new network is created and being used, a Virtual Router is automatically launched to support these features. Since the Virtual Router is a Single Point of Failure, you should turn on the HA (High Available) option; which actually addes a 2nd Virtual Router on each network. While this is pretty cool, it makes the number of System VM’s go up and when you don’t need them it’s kind of wasting resources.

Fixing ‘Clocksource tsc unstable’ warnings on older hardware that result in a system crash

Sat, 10 Mar 2012 16:02:00 +0100

We still have some older hardware running for non-critical and lightweight operations. Since upgrading these old boxes to Debian Squeeze weird things started to happen. After some time, the system clock stops working and that results in processes to crash and new ones unable to start. Of course, nothing works properly on any computer without a working system clock. Here you see how that looks like:

First thought was to replace the CMOS battery, but that didn’t help and it occurred on multiple machines around the same time. Rebooting solves the problem for a while but 24-36 hours later the system clock stops again. Sometimes the problem stays away much longer. Also, some servers with the same hardware do not seem to have the problem. It seems hardware related. But is that really true?

HOWTO backup an OpenLDAP server

Wed, 07 Mar 2012 18:49:51 +0100

In a previous post I described howto restore a OpenLDAP server from backup . But how to backup Open LDAP?

The backups I make consist of two parts:

1. First backup the LDAP database itself using a program called ‘slapcat.’ Slapcat is used to generate an LDAP Directory Interchange Format (LDIF) output based upon the contents of a given LDAP database. This is a text version of your database which can be imported later. Think of it as a SQL-backup for relational databases. Anyway, here’s how to run slapcat on the OpenLDAP server:

Loads of sleeping MySQL connections?

Mon, 05 Mar 2012 16:08:24 +0100

Then probably the code is not properly disconnecting the MySQL connection. The time MySQL waits for the next command to be send, can be controlled by the ‘wait_timeout’ parameter. The default is a massive 28800 seconds (8h). Setting this value too low gives ‘MySQl server has gone away’ errors, so be careful.

Debugging scripts that create many ‘sleeping’ connections is challenging. An easy solution is setting the ‘wait_timeout’ parameter to something low after the MySQL connection is started. This setting is then valid only for this current session.

HOWTO solve OpenLDAP bdb_equality_candidates errors

Mon, 05 Mar 2012 11:33:18 +0100

After restoring an OpenLDAP server I found these lines in the logs:

Mar  5 06:50:03 ldap slapd[4815]: <= bdb_equality_candidates: (uidNumber) index_param failed (13)
Mar  5 06:50:04 ldap slapd[4815]: <= bdb_equality_candidates: (uid) index_param failed (13)

This means OpenLDAP is query’ing its database, but found no index for fields it often uses. In this case ‘uid’ and ‘uidNumber’. It seems due to restoring the backup, these indexes got lost. Here is how to add the indexes again:

HOWTO restore an OpenLDAP server from backup

Mon, 05 Mar 2012 11:32:01 +0100

Just a quick instruction on howto restore an OpenLDAP server using a ‘ldif’-backup-file:

1. Setup the server

2. Configure the ‘slapd’ package, be sure to use the right database name. It’s a bit confusing: you enter it as ldaptree.company.com, but this will be used to create an empty database. Make sure it matches your backup ldif structure.

3. Make sure OpenLDAP is stopped

4. Read the backupdata into the OpenLDAP database using slapadd program

Allowing user processes to run for 10 minutes max on Linux – a first approach

Sun, 04 Mar 2012 19:19:23 +0100

Perl processes that got stock kept me from sleep last night. I’m not sure what happened, but probably they were waiting for the database that reached a max_connections limit for this particular user. When Apache reached its max processes, monitoring paged me. Fortunately this is all running in a cluster, so no service was interrupted.

A few days ago I found myself killing Apache processes from a user that created an endless loop. Also, some scripts our users upload generate a lot of load (which is ok for a while, but not too long). So I thought I’d write something in bash to help me with all this 🙂

HOWTO quickly add a route in Mac OSX

Sun, 04 Mar 2012 17:27:03 +0100

Adding a route manually can be necessary sometimes. When on Linux, I know the command by head:

sudo route add -net 10.67.0.0/16 gw 192.168.120.254

On the Mac the command is similar, but a bit different 🙂 Just as a note to myself and anyone else interested:

sudo route -n add -net 10.67.0.0/16  192.168.120.254

This sets up a route to the 10.67.0.0/16 net through gateway 192.168.120.254. First one on Linux, second one on Mac OSX.

CloudStack 3.0 released today!

Tue, 28 Feb 2012 23:05:53 +0100

Good news today when I came home from the office: CloudStack 3.0 stable has been released! I’d really been looking forward to this, because we’ll be using it in a big cloud project at work. With this release, we can finalize our cloud design and proof-of-concept setup. More about that later.

Thanks to the CloudStack team for this great release! 🙂

Redirecting incoming tcp connections with redir

Tue, 28 Feb 2012 22:58:27 +0100

Sometimes you need an easy way to redirect incoming connections to another system. For example when migrating an old box to a new one. Today I came across an old note from myself explaining this. I thought it might be worth sharing 🙂

You’ll need the ‘redir’ program for this to work:

apt-get install redir

Redir redirects tcp connections coming in to a local port to a specified address/port combination like this:

CloudStack from an end-user’s point of view

Sat, 25 Feb 2012 00:17:03 +0100

For a Cloud project at work, I made some screenshots of CloudStack to show our management. They wanted to know what an end-user will see in CloudStack 3.0. This is CloudStack from an end-user’s point-of-view:

The login screen is where it all starts..

After logging in, users are presented with a Dashboard. Here is a nice and clear overview of what VM’s are running and stopped. As you can see, there is a nice menu at the right.

Backuping using rsnapshot made my life easier

Sat, 18 Feb 2012 13:18:25 +0100

As a sysadmin I’ve many things to take care of. One of the most important is backups. As websites and mailarchives become larger and lager, it is an ongoing challenge to fit as many backups in the available backup space.

In the early days we’re backupping using rsync, tar and gzip. The biggest drawback was it takes a lot of space. On the bright side, it’s plain simple and just always works. All you’ve to do is untar an archive and everything is there again (i.e.: happy customer!). It helped me on many occasions. So I kept this old method for a long time and looked around for alternatives.

Home-key in OSX-Terminal

Sat, 18 Feb 2012 12:27:33 +0100

I’m working in Terminal a lot. Lately I more and more use my MacBook Pro instead of my Ubuntu desktop. I’d to overcome some small issues. One was the missing Home-key, which allows you to jump all the way to the left on the current line.

Although the Mac doesn’t have a Home key, its function is available by pressing Fn+Shift+Left Arrow. And even better: Fn+Shift+Right Arrow jumps all the way to the right.

Impressed by upcoming CloudStack 3.0

Sun, 12 Feb 2012 13:35:23 +0100

For about 6 weeks we’ve been working on finding out if -and how- it’d be possible to create our own cloud. My colleague Pim did a very good job on sorting out all the different software solutions (of course Open Source) and came up with CloudStack as the one we’d definitely to test with. So we did!

We’ve tested both the current stable 2.2 release and the upcoming 3.0 release, which is currently in beta. It took us quite some time to get the right hardware to test with. At first we used Ubuntu as OS but that turned out to be the wrong choice – for now. Ubuntu just isn’t very well supported and CloudStack more or less wants you to use RHEL or one of its free alternatives, like CentOS. We wanted to use Ubuntu at first because we had a lot of Debian experience. Although Google was our friend in helping sorting out differences between Debian-style and RedHat-style (for example in the networking setup). Looking back, moving to CentOS was no problem at all. We even have Kickstart running to be able to do quick unattended installs for the compute nodes. Cool! By the way, this is about the OS on the Compute nodes and Management-server. VM’s can of course be of any kind. In our case they definitely will run Debian.

This is what I’ve learned @ “Build an Open Source Cloud Day”

Sat, 04 Feb 2012 21:14:50 +0100

Going to Antwerp by train

Thursday February 2nd me and colleague Pim went to Antwerp by train to attend “Build an Open Source Cloud“-Day, hosted by INUITS the following day.

The programme looked promising and I really looked forward to meeting David Nalley and Mark Hinkle from Citrix’ CloudStack. The last months we kept an eye on CloudStack: tested their current 2.2 release, and the 3.0 beta’s. Although we had CloudStack more or less up and running, there were still many questions to ask and many things to learn. This was a perfect opportunity for that.

Alt-key in OSX-Terminal

Mon, 30 Jan 2012 21:42:02 +0100

By default, the Alt-key doesn’t work in Mac OSX-Terminal. Today I found a simple solution for this problem:

Terminal > Preferences > Settings > Keyboard
Just check ‘Use option as meta key’ at the bottom and you’re good to go 🙂

Voorbereiding op zonnepanelen: 40% besparing op stroom door Wattcher en energiemeter

Fri, 02 Dec 2011 20:22:27 +0100

In augustus 2011 was het stroomverbruik van ons gezin (2 volwassenen en een zoon van bijna 1) 297kWh. Al enige tijd was ik de standen aan het bijhouden en zo’n 300kWh per maand vond ik simpelweg teveel. Twee dingen hebben geholpen om dit verbruik in 3 maanden tijd terug te brengen naar zo’n 180kWh: een besparing van 40%!

Dit is hoe we dat gedaan hebben:
1. Inzicht krijgen in hoeveel “sluipverbruik” er is
Hiervoor hebben we een Wattcher aangeschaft: dit apparaat geeft op 10 watt nauwkeurig aan hoeveel stroom er op een bepaald moment gebruikt wordt. Dus wordt er ‘s avonds als je op het punt staat naar bed te gaan nog 400 watt gebruikt dan weet je dat als je 7 uur slaapt je ‘s ochtends 7×400 = 2,8kWh verder bent. En dan heb je alleen geslapen 😉

Collectd met Collectd Graph Panel: mooi en overzichtelijke performance statistieken

Tue, 29 Nov 2011 15:08:41 +0100

Het leuke van publiceren waar je mee bezig bent is dat je feedback krijgt. Dat gebeurde ook met mijn korte stukje over Dstat gisteren. Reacties van een oud collega en collega beheerder hier uit Leiden. Leuk!

Pim tipte me Collectd: een daemon die allerlei statistieken verzamelt over een systeem en dit opslaat in RRD files. Helemaal leuk werd het toen hij erbij vertelde dat hij er zelf een front-end bij gemaakt heeft: Collectd Graph Panel (OpenSource). Gaaf!

About Remi Bergsma

Mon, 28 Nov 2011 10:08:28 +0000

I’m a Linux Engineer since 1997 with expertise in building and managing infrastructure (based on Open Source software) and Linux environments. I’ve broad experience in Debian, CentOS, Red Hat and SUSE Linux distributions and are certified for most of them as well (RHCE/CLE/LPIC-3).

My goal is to automate what can be automated, so that everything works rock solid and I have the time to explore new techniques and tools (that might result in even better automation).