Here are the slides from my presentation at the CloudStack Collaboration Conference in Denver:
Archives For 30 November 1999
As you might have noticed, I switched jobs on Feb 1st and was quite busy after that. I joined Schuberg Philis, a Mission Critical Outsourcing company in The Netherlands. I met the folks on the CloudStack Collaboration Conference in Amsterdam, back in November. I have a very cool job, as I work with a bunch of very skilled colleagues on our Mission Critical Cloud, that is powered by Apache CloudStack.
The last two months I spent quite some time working with CloudStack, debugging issues and most of all: building new capacity and migrating old so-called PODs to new ones. Recently, CloudStack 4.3 was released and I believe this is the best release ever. Feels “snappier”, has lots of fixes and new cool features. We upgraded our pre-production cloud (running ~600 VMs) and apart from one little issue everything went smooth. Impressed!
Next week, I’ll be in Denver, CO at the CloudStack Collaboration Conference. I’ll do a talk and explain why you should use configuration management and will show you how to implement configuration management in your existing environment using 5 steps. This can be quite challenging, because when a lot of manual work is done, people are usually very busy. How do you find the time to implement configuration management?
Meet me in Denver on April 9-11!
I’ve been working a lot with CloudStack Advanced networking and find it very flexible. Recently, I had another opportunity to test its flexibility when a customer called: “We want VM’s in your CloudStack cloud, but these VM’s are only allowed be reachable from our office, and not from the public internet”. Firewalling? No, they required us to use their VPN solution.
Is CloudStack flexible enough for this to work? Yes, it is. In this blog I’ll tell you how we did it. And it doesn’t even matter what VPN device you use. This will work regardless of brand and features, as lang as it supports a public ip-address to connect over the internet to another VPN device, and has a private network behind it. All VPN devices I know of support these basic features.
VPN (Virtual Private Networking)
The client’s office is connected to the internet and has a VPN device. We received another device as well to host in our data center and the two talk to each other over the public internet in a secure way. Probably speaking IPsec or similar but that is beyond the scope of this blog.
The VPN device in the data center has a public ip-address on its WAN port but also has some ports for the internal network. We configured it to use the same network CIDR as we did in the CloudStack network we created for this customer. Let’s use 10.10.16.0/24 as an example in this blog. And now the problem: this cloud network is a tagged network and the VPN device we received is not VLAN-capable.
VLANs in the Cloud
CloudStack Advanded networking relies on VLANs. Every VLAN has its own unique ID. Switches use this VLAN ID to keep the VLAN networks apart and make sure they’re isolated from each other. Most switches support VLANs as well, and that’s were we’ll find the solution to this problem.
Configuring the switch
We connected the VPN device to our switch and set its port to UNTAGGED for the VLAN ID the CloudStack network uses. In other words, devices connected to this port now do not need to know about the VLAN. The switch will add it as traffic flows. This means the VPN device will use an ip-address in the 10.10.16.0/24 range and is able to communicate with the VM’s in the same network. The CloudStack compute nodes have their switch ports set to TAGGED and the switch makes communication between them possible.
Overview of ip-addresses:
- 10.10.16.1 – the internal VPN device ip-address in the data center
- 10.10.16.11 – the first VM’s ip-address
- 10.10.16.12 – the second VM’s ip-address
The VM’s have their default gateway set to the VPN device’s 10.10.16.1 address. Also, the office needs to be configured in a way it knows the 10.10.16.0/24 network is handled by the VPN device located there. Users in the office will now be able to access the VM’s on the 10.10.16.0/24 network.
Conclusion
While the VM’s are hosted on our CloudStack cloud on the internet, they do not have public ip-addresses and thus are not reachable. The only public ip-address for this customer is the one configured on the VPN device in the data center. This provides the same level of security as you’d have with physical servers but adds the power of a cloud solution.
Thanks to the flexibility of the CloudStack Advanced Networking this cloud be done!
It just came to my attention that a vulnerability in Apache CloudStack was discovered, as John Kinsella writes in his post to the Apache CloudStack dev-mailinglist.
A malicious user could, for example, delete all VMs in the system. Addressing this issue is especially important for anybody using CloudStack in a public environment.
The vulnerability report has an easy work-around that I will mention here as well:
mysql -p -u cloud -h mgt-server-ip update cloud.user set password=RAND() where id=1; \q
Hugo Trippaers of Schuberg Philis discovered this issue. Thanks for sharing!
I’ve been building redundant storage solutions for years. At first, I used it for our webcluster storage. Nowadays it’s the base of our CloudStack Cloud-storage. If you ask me, the best way to create a redundant pair of Linux storage servers using Open Source software, is to use DRBD. Over the years it has proven to be rock solid to me.
DRBD is a Distributed Replicated Block Device. You can think of DRBD as RAID-1 between two servers. Data is mirrored from the primary to the secondary server. When the primary fails, the secondary takes over and all services remain online. DRBD provides tools for failover but it does not handled the actual failover. Cluster management software like Heartbeat and PaceMaker are made for this.
In this post I’ll show you how to install and configure DRBD, create file systems using LVM2 on top of the DRBD device, serve the file systems using NFS and manage the cluster using Heartbeat.
Installing and configuring DRBD
I’m using mostly Debian so I’ll focus on this OS. I did setup DRBD on CentOS as well. You need to use the ELREPO repository to find the right packages.
Squeeze-backports has a newer version of DRBD. If you, like me, want to use this version instead of the one in Squeeze itself, use this method to do so:
echo " deb http://ftp.debian.org/debian-backports squeeze-backports main contrib non-free " >> /etc/apt/sources.list echo "Package: drbd8-utils Pin: release n=squeeze-backports Pin-Priority: 900 " > /etc/apt/preferences.d/drbd
Then install the DRBD utils:
apt-get update apt-get install drbd8-utils
As the DRBD-servers work closely together, it’s important to keep the time synchronised. Install a NTP system for this job.
apt-get install ntp ntpdate
You also need a kernel module but that one is in the stock Debian kernel. If you’re compiling kernels yourself, make sure to include this module. When you’re ready, load the module:
modprobe drbd
Verify if all went well by checking the active modules:
lsmod | grep drbd
The expected output is something like:
drbd 191530 4 lru_cache 12880 1 drbd cn 12933 1 drbd
Most online tutorials instruct you to edit ‘/etc/drbd.conf’. I’d suggest not to touch that file and create one in /etc/drbd.d/ instead. That way, your changes are never overwritten and it’s clear what local changed you made.
vim /etc/drbd.d/redundantstorage.res
Enter this configuration:
resource redundantstorage {
protocol C;
startup { wfc-timeout 0; degr-wfc-timeout 120; }
disk { on-io-error detach; }
on storage-server0.example.org {
device /dev/drbd0;
disk /dev/sda3;
meta-disk internal;
address 10.10.0.86:7788;
}
on storage-server1.example.org {
device /dev/drbd0;
disk /dev/sda3;
meta-disk internal;
address 10.10.0.88:7788;
}
}
Make sure your hostnames match the hostnames in this config file as it will not work otherwise. To see the current hostname, run:
uname -n
Modify /etc/hosts, /etc/resolv.conf and/or /etc/hostname to your needs and do not continue until the actual hostname matches the one you set in the configuration above.
Also, make sure you did all the steps so far on both servers.
It’s now time to initialise the DRBD device:
drbdadm create-md redundantstorage drbdadm up redundantstorage drbdadm attach redundantstorage drbdadm syncer redundantstorage drbdadm connect redundantstorage
Run this on the primary server only:
drbdadm -- --overwrite-data-of-peer primary redundantstorage
Monitor the progress:
cat /proc/drbd
Start the DRBD service on both servers:
service drbd start
You now have a raw block device on /dev/drbd0 that is synced from the primary to the secondary server.
Using the DRBD device
Let’s create a filesystem on our new DRBD device. I prefer using LVM since that makes it easy to manage the partitions later on. But you may also simply use the /dev/drbd0 device as any block device on its own.
Initialize LVM2:
pvcreate /dev/drbd0 pvdisplay vgcreate redundantstorage /dev/drbd0
We now have a LVM2 volume group called ‘redundantstorage’ on device /dev/drbd0
Create the desired LVM partitions on it like this:
lvcreate -L 1T -n web_files redundantstorage lvcreate -L 250G -n other_files redundantstorage
The partitions you create are named like the volume group. You can now use ‘/dev/redundantstorage/web_files’ and ‘/dev/redundantstorage/other_files’ like you’d otherwise use ‘/dev/sda3’ etc.
Before we can actually use them, we need to create a file system on top:
mkfs.ext4 /dev/redundantstorage/web_files mkfs.ext4 /dev/redundantstorage/other_files
Finally, mount the file systems:
mkdir /redundantstorage/web_files mkdir /redundantstorage/other_files mount /dev/redundantstorage/web_files /redundantstorage/web_files mount /dev/redundantstorage/other_files /redundantstorage/other_files
Using the DRBD file systems
Two more steps are needed to set up before we can test our new redundant storage cluster: Heartbeat to manage the cluster and NFS to make use of it. Let’s start with NFS, so Heartbeat will be able to manage that late on as well.
To install NFS server, simply run:
apt-get install nfs-kernel-server
Then setup what folders you want to export using your NFS server.
vim /etc/exports
And enter this configuration:
/redundantstorage/web_files 10.10.0.0/24(rw,async,no_root_squash,no_subtree_check,fsid=1) /redundantstorage/other_files 10.10.0.0/24(rw,async,no_root_squash,no_subtree_check,fsid=2)
Important:
Pay attention to the the ‘fsid’ parameter. It is really important because it tells the clients that the file system on the primary and secondary are both the same. If you omit this parameter, the clients will ‘hang’ and wait for the old primary to come back online after a fail over happens. Since this is not what we want, we need to tell the clients the other server is simply the same. Fail-over will then happen almost without notice. Most tutorials I read do not tell you about this crucial step.
Make sure you have all this setup on both servers. Since we want Heartbeat to manage our NFS server, we need not to start NFS on boot. To do that, run:
update-rc.d -f nfs-common remove update-rc.d -f nfs-kernel-server remove
Basic Heartbeat configuration
Install the heartbeat packages is simple:
apt-get install heartbeat
If you’re on CentOS, have a look at the EPEL repository. I’ve successfully setup Heartbeat with those packages as well.
To configure Heartbeat:
vim /etc/ha.d/ha.cf
Enter this configuration:
autojoin none auto_failback off keepalive 2 warntime 5 deadtime 10 initdead 20 bcast eth0 node storage-server0.example.org node storage-server1.example.org logfile /var/log/heartbeat-log debugfile /var/log/heartbeat-debug
I set ‘auto_failback’ to off, since I do not want another fail-over when the old primary comes back. If your primary server has better hardware than the secondary one, you may want to set this to ‘on’ instead.
The parameter ‘deadtime’ tells Heartbeat to declare the other node dead after this many seconds. Heartbeat will send a heartbeat every ‘keepalive’ number of seconds.
Protect your heartbeat setup with a password:
echo "auth 3 3 md5 your_secret_password " > /etc/ha.d/authkeys chmod 600 /etc/heartbeat/authkeys
You need to select an ip-address that will be your ‘service’-address. Both servers have their own 10.10.0.x ip-address, so choose another one in the same range. I use 10.10.0.10 in this example. Why we need this? Simply because you cannot know to which server you should connect. That’s why we will instruct Heartbeat to manage an extra ip-address and make that alive on the current primary server. When clients connect to this ip-address it will always work.
In the ‘haresources’ file you describe all services Heartbeat manages. In our case, these services are:
– service ip-address
– DRBD disk
– LVM2 service
– Two filesystems
– NFS daemons
Enter them in the order they need to start. When shutting down, Heartbeat will run them in reversed order.
vim /etc/ha.d/haresources
Enter this configuration:
storage-server0.example.org \ IPaddr::10.10.0.10/24/eth0 \ drbddisk::redundantstorage \ lvm2 \ Filesystem::/dev/redundantstorage/web_files::/redundantstorage/web_files::ext4::nosuid,usrquota,noatime \ Filesystem::/dev/redundantstorage/other_files::/redundantstorage/other_files::ext4::nosuid,usrquota,noatime \ nfs-common \ nfs-kernel-server
Use the same Heartbeat configuration on both servers. In the ‘haresources’ file you specify one of the nodes to be the primary. In our case it’s ‘storage-server0’. When this server is or becomes unavailable, Heartbeat will start the services it knows on the other node, ‘storage-server1’ in this case (as specified in the ha.cf config file).
Wrapping up
DRBD combined with Heartbeat and NFS creates a powerful, redundant storage solution all based on Open Source software. When using the right hardware you will be able to achieve great performance with this setup as well. Think about RAID controllers with SSD-cache and don’t forget the Battery Backup Unit so you can enable the Write Back Cache.
Enjoy building your redundant storage!
Remi Bergsma's blog 