Category: Linux Sysadmin

77 posts

Back in June, just before I went off for holiday, I attended a CFEngine training in Amsterdam. When I returned from holiday a few weeks later, me and my team started making plans to implement CFEngine in our environment. After two months of hard work, I’m proud to say we manage about 350 out of our 400 Linux servers with CFEngine!

The ride has been fun, although not always easy. In this post I’ll give a quick overview of our CFEngine implementation, where I found useful info, etc.

CFEngine is different
To start, let me tell you that one of the most difficult parts of learning CFEngine is to get used to the terminology and to ‘think’ CFEngine. For example, a ‘class’ in CFEngine is not what you think it is. It has nothing to do with object oriented programming. It’s more like a ‘context’ that you can use to make decisions. There’s no ‘flow control’ in CFEngine either: no IF/THEN/ELSE, no FOR/FOREACH/WHILE etcetera. In CFEngine classes are used for decision making, and, since CFEngine is smart, it does looping automatically. This results in clean and easy-to-read code.

CFEngine works on top of a theoretical model called ‘Promise Theory‘ by Mark Burgess (author of CFEngine). This theory models the behavior of autonomous agents in an environment without central authority, based on only promises of behavior made by each agent, and shows that even without central control , the system can converge to a stable state.

To get used to it, read ‘Learning CFEngine 3‘ by Diego Zamboni, as it will walk you through all of it with a lot of examples. The quote above is also from the book.

The basic idea is that each agent makes promises only about its own behavior, since that is all it can control. In CFEngine 3, everything is a promise.

Examples:
– a file promises to have a certain content and to be executable
– a service promises to be running
– a user account promises to exist (or not to exist) and have certain properties

When CFEngine finds a promise is not kept, it will do everything it knows about to make the promise true. If it cannot reach the promised state at first, it tries to the next best. Over time, the system converges to the desired (promised) state.

Once you get it and get used to it, it actually makes sense and is pretty easy to implement.

With great power comes great responsibility
This one-liner says it all. When you have a configuration management system that manages a lot of servers, you better be careful what promises you have it keep. This is why you need to manage CFEngine promises like software. You need version control and it needs to be flexible as well. I’ve read a lot about this subject and I believe Git is the way to go. This blog by Brian Bennett pretty much nails it. I got a lot of inspiration from it, thanks Brian!

I implemented these ‘branches’ in Git:
– development (aka master)
– beta
– pre-production
– production

This works perfectly: develop new promises in the ‘development’ branch, then merge to ‘beta’ branch to test on some of our own test servers. When everything works together and seems stable, we merge to the next branch ‘pre-production’. This is then tested on ~15 real production servers so it better be good. But when it isn’t, the impact is still not too high and it should be fixed before it ever hits ‘production’. Production branch is everything that is stable and is used on all ~350 servers.

Every time we merge to either ‘pre-production’ or ‘production’, we create a Git ‘tag’ with a date, that allows for easy roll backs. Whenever we need to get back to a certain state, we can always just checkout a tag. This is also very useful for audit trails, by the way.

Actually, we’re using another branch called ‘hotfix’. Whenever there’s an emergency to fix, we branch a ‘hotfix’ from ‘production’ and do the fix. This is for example when a promise misbehaves. This branch is then merged to production when ready, and also to ‘development’. Git handles this nicely: whenever the hotfix makes it all the way from ‘development’ to ‘production’, Git recognizes this commit was already processed earlier and ignores it.

Git commits, branches and tags in CFEngine repo
Git commits, branches and tags in CFEngine repo

 

This is a screenshot from ‘Gitk’ that shows the commits, branches (green) and tags (yellow). As you can see, ‘production’ and ‘pre-production’ are at the same level now, so nothing new is tested in ‘pre-production’ at the moment. Quite some work is tested in the ‘beta’ branch and there are already some fixes committed in ‘master’. Recently there was a ‘hotfix’ branch that has now been merged. It should give an idea of how it works. It provides a clear overview and we now know about every change on the configuration of our servers. Clicking on a commit show what changed, who did it, etc.

CFEngine Policy hubs
For each of the 4 branches we’ve created a CFEngine policy hub. The policy hub is a server running CFEngine software that serves the given branch to the agents (the Linux servers connected to it). Linux servers can even switch between them by ‘bootstrapping’ to one of the 4 policy hubs. Although we only use that on our test servers.

Manage what’s ‘in flight’ with a CFEngine Trello board
Trello provides an intuitive and modern web interface that allows you to manage ‘cards’ on different ‘lists’ on a ‘board’. To get an idea, see the example Trello board below (click on it to enlarge).

Trello CFEngine board
Trello CFEngine board

 

New cards are usually created in ‘Feature requests’ or ‘Bugs’ and then transferred to ‘Working on it!’. The number of cards in this stage should be limited, as you can only work on a number of things at the same time. This is actually Kanban style. Next, we’ve created a list for each Git ‘branch’ we have and cards flow from ‘beta’ to ‘pre-production’ and finally ‘production’. Moving cards is just dragging & dropping. Each month, cards in ‘production’ are archived. This creates an overview of what new work is to be done (‘Feature requests’ and ‘Bugs’), what we’re currently working on and what’s in each of the branches. Trello has the overview, Git has the code and the details. Also, Trello is perfect for communication between team members. Notes, comments, documents, lists, etcetera can all be created with ease.

Testing promises
To be able to test the promises on our local laptops, we’re using a tool called Vagrant. Vagrant sets up Virtual Machines (for example using Virtual Box) and allows you to ‘destroy’ and ‘create’ them within minutes. All team members have a local Git checkout, that is also available in the Vagrant boxes. This allows us to test any change before even committing. We have Vagrant boxes setup for all Linux distributions we support. It’s so easy and so fast to test changes that everybody does. And even when an error slips through, other team members will soon notice and it’s usually fixed within minutes, before it ever hits the ‘beta’ branch.

Bugs
We encountered a strange bug when using SLES 11 and CFEngine 3.5: CFEngine (community edition) got running with the ‘SIGPIPE’ signal blocked. When CFEngine restarts SSH, this too gets running with ‘SIGPIPE’ blocked. This results in ‘sudo’ no longer working. It would just return nothing at all. It took us quite some time to figure out it was the ‘SIGPIPE’ signal that was blocked. The root cause probably lies in an old ‘Bash’ version (3.51) that SLES uses, combined with something CFEngine triggers. We’ve now implemented an automated work-around (made a CFEngine promise) that fixes the problem. We did some nice team work on this one!

Conclusion
CFEngine’s learning curve might be steep, but the result is definitely rewarding. Combined with Git and Trello it allows for fine control and great overview of configuration changes. Our whole team is involved in changes, they are reviewed and result in high quality code. This eventually makes the Linux servers we manage more stable. Also, it’s a great feeling to be in-control and know what’s going on our servers.

From this point on, we’ll continue to both scale horizontally (add more servers) and vertically (add more promises). After two months of daily working with CFEngine, I’ve to say I really like it and I enjoy writing promises.

I’ll keep you posted, I promise 😉

Data security is getting more and more important these days. Imaging you work as a sysadmin, on a laptop, and it gets lost. Of course you might lose your work (but you have a backup, right?). The real problem: your sensible data (SSH private key for example) is no longer under your control.

In this blog I explain how you can add an encrypted partition to Linux. As long as you also use a password protected screensaver, with a decent password (to protect a running, logged in laptop), no one can access your data. Even rebooting into single user mode (to by-pass the login screen) won’t help. No access to the encrypted disk without a working passphrase. Wiping your disk and reinstall is an option, but your data is not unveiled.

LUKS: Linux Unified Key Setup
Linux ships with LUKS, that is short for ‘Linux Unified Key Setup’. It’s a tool and technique to setup encrypted devices. This device can be a laptop harddisk, but also a USB-pendrive or a virtual disk (when you have a virtualized server). Encryption with LUKS works on the block level, so filesystems above it are not even aware. This is nice, because it also means you can use LVM inside a LUKS encrypted block device. The encrypted drive is protected with a passphrase that you need to enter at boot.

Below I’ll show you how to set it up. Be aware we are formatting partitions. In other words, you will lose all data on the partition you will experiment with. For testing purposes, use a USB-pendrive or a spare partition. I’m using a virtual disk called /dev/sdc for this demo.

Note: you need the kernel module dm_crypt loaded for this to work.

modprobe dm_crypt

Don’t forget to make it persistent after a reboot.

To format the partition:

cryptsetup luksFormat /dev/sdc

Output:

WARNING!
========
This will overwrite data on /dev/sdc irrevocably.

Are you sure? (Type uppercase yes): YES Enter LUKS passphrase: Verify passphrase:

Now that the encrypted disk is created, we’ll open it:

cryptsetup luksOpen /dev/sdc encrypted_disk

Output:

Enter passphrase for /dev/sdc:

To unlock the disk, you need to enter the passphrase you just set. You need to do this every time you want to unlock the disk.

The ‘encrypted_disk‘ part of the command above, is used to map a device name to your encrypted disk. In this case: ‘/dev/mapper/encrypted_disk’ is created. So, the encrypted disk called ‘/dev/sdc’ now has an extra name to refer to its unlocked state called ‘/dev/mapper/encrypted_disk’. This device is also a block device, on which you could run ‘fdisk’ or ‘mkfs.ext4’, etc.

But stop, before you do that. When you’d like to use multiple partitions, I’d suggest using LVM inside the encrypted disk. This isn’t visible from outside; only when you’ve unlocked it the LVM partitions appear. It prevents from entering a password for each and every encrypted disk you create. Also, LVM is more flexible in resizing its logical volumes.

Let me show you how to setup LVM on the encrypted disk:

pvcreate /dev/mapper/encrypted_disk

Output:

Physical volume "/dev/mapper/encrypted_disk" successfully created 
vgcreate encrypted /dev/mapper/encrypted_disk

Output:

Volume group "encrypted" successfully created
lvcreate encrypted --name disk1 --size 10G

Output:

Logical volume "disk1" created
lvcreate encrypted --name disk2 --size 10G

Output:

Logical volume "disk2" created

You now have two more block devices called ‘/dev/encrypted/disk1’ and ‘/dev/encrypted/disk2’. Let’s put a file system on top of them:

mkfs.ext4 -m0 /dev/encrypted/disk1
mkfs.ext4 -m0 /dev/encrypted/disk2

The two encrypted partitions are now ready to be used. Let’s mount them somewhere:

mkdir -p /mnt/disk1 /mnt/disk2
mount /dev/encrypted/disk1 /mnt/disk1
mount /dev/encrypted/disk2 /mnt/disk2

This works all pretty cool already. But when you reboot, you’ll have to run:

cryptsetup luksOpen /dev/sdc encrypted_disk
vgscan
lvchange --activate y encrypted/disk1
lvchange --activate y encrypted/disk2
mount /dev/encrypted/disk1 /mnt/disk1
mount /dev/encrypted/disk2 /mnt/disk2

Entering the passphrase is required after the first command. Line 2 scans for new LVM devices (because when unlocking the encrypted device, a new block device appears). Line 3 and 4 activate the two logical volumes, and finally they are mounted.

Automating these steps
It is possible to automate this. That is, Linux will then ask for the passphrase at boot time and mount everything for you. Just think about that for a while. When booting a laptop this is probably what you want. But if it is a server in a remote location, it might not, as you need to enter it on the (virtual) console for it to continue booting. No SSH access at that time.

Anyway, you need to do two things. First is to tell Linux to unlock the encrypted device at boot time. Second is to mount the logical volumes.

To start, lookup the UUID of the encrypted disk, /dev/sdc:

cryptsetup luksDump /dev/sdc | grep UUID

The result should be something like:

UUID: b8f60c1d-ffeb-4aaf-8368-9e5d4d29fc52

Open /etc/crypttab and enter this line:

encrypted_disk UUID=b8f60c1d-ffeb-4aaf-8368-9e5d4c29fc52 none

The first field is the name of the device that is created, use the same as when using luksOpen above. The second field is the UUID we just found. The final field is the password, but this should be set to ‘none‘ as this prompts for the passphrase. Entering the passphrase in this file is a bad idea, if you ask me.

Final step is to setup /etc/fstab to mount the encrypted disks automatically. Add these lines:

/dev/encrypted/disk1 /mnt/disk1 ext4 defaults 0 0
/dev/encrypted/disk2 /mnt/disk2 ext4 defaults 0 0

I’m using device names here, because LVM gives me the /dev/encrypted/disk[12] name every time. When you did not use LVM, it’s probably wise to use UUID’s in /etc/fstab instead. This makes sure the right filesystem is mounted, regardless of the device’s name.

Time to reboot. During this reboot, Linux will ask for the passphrase of the /dev/sdc device. On RHEL 6 it looks like:

RHEL 6 asks for the LUKS passphrase of /dev/sdc during boot
RHEL 6 asks for the LUKS passphrase of /dev/sdc during boot

It might look different on your OS. The Ubuntu version looks a bit prettier, for example.

Enter the passphrase, hit enter, and Linux should continue booting normally. Then login to the console (or SSH) and verify if the two disks are mounted.

...
/dev/mapper/encrypted-disk1 on /mnt/disk1 type ext4 (rw)
/dev/mapper/encrypted-disk2 on /mnt/disk2 type ext4 (rw)
...

Ubuntu makes this very easy to setup: they just have a checkbox during install that says ‘encrypt disk’ and will setup LUKS. But, you end up with everything in one big / partition without LVM. That’s why I prefer to configure it myself, and with these instructions so can you.

Conclusion
It is cool that these security features are now mainstream and easy to use. Do yourself a favor, and setup LUKS today!

Today I started learning CFEngine 3

— 787 views — 1 Comment

Me and two colleagues went to Amsterdam today, for a 1-day CFEngine-3 training. I’ve worked with configuration management before (Puppet), and my goal is to explore alternatives to be able to pick the right tool. Today’s quick overview training was a nice opportunity to get into CFEngine and meet some people behind the scenes.

Impression of CFEngine training
Impression of CFEngine training

 

Diego Zamboni, author of the book Learning CFEngine 3, taught us what the concepts behind CFEngine are, how the language is build and how to get started. He demo’ed a lot of things and answered all of our questions. It was a very informative training that really inspired me.

What surprised me most, was that CFEngine is actually a pretty nice monitoring tool! A smart one, because it is either able to fix things, or is able to report it. All in all, I’ve to say I’m really impressed by CFEngine 3.5. We have now inspiration to create a good plan and then start working on implementing it. Looking forward to it!

Thanks Diego and Carsten for this training and the nice drinks we had afterwards 🙂

LPICLevel3_logoToday I passed the senior level LPIC-301 core exam. I am now LPIC-3 certified!

The exam objectives are OpenLDAP and Capacity planning. Both of which I used in past projects, so I was already pretty familiar with them. In my current job I came across AD integration and Kerberos, and those were objectives as well.

In the past weeks I went through every detail of the objectives and learned some new stuff. My score on the exam was 690/800.

I’ve mainly used the IBM Developer Works LPIC-3 website, OpenLDAP manual and the LPIC-3 page hosted at rootkit.nl. As always, the man pages of the Linux commands are very useful.

After studying the materials, it’s just practice, practice, practice. I’ve setup an environment in which I could test and play with the tools and that worked really well for me.

The Security LPIC-3 specialty exam is the next one on my wish-list. I’ve already started to study it because I’m really interested in these topics.

Later this year I’ll be doing the RedHat RHCSA and RHCE exams. I’ve booked them on the same day, so that’s probably getting a bit touch 😉 But I’m looking forward to it and enjoy the steps on the certification road.

As discussed in a previous post, multi-factor authentication really makes things more secure. Let’s see how we can secure services like SSH and the Gnome desktop with multi-factor authentication.

The Google Authenticator project provides a PAM module than can be integrated with your Linux server or desktop. This PAM module is designed for for home and other small environments. There is no central management of keys, and the configuration is saved in each users home folder. I’ve successfully deployed this on my home server (a Raspberry Pi) and on my work laptop.

When using a Debian-like OS, you can install it with a one-liner

apt-get install libpam-google-authenticator

But note the packaged version is old and does not support all documented options. Below I talk about the ‘nullok’ option, but that is not supported in the packaged version. You then see this error:

sshd(pam_google_authenticator)[884]: Unrecognized option "nullok"
sshd(pam_google_authenticator)[887]: Failed to read "/home/pi/.google_authenticator"

That’s why I suggest building from source, as this can be done quickly:


apt-get remove libpam-google-authenticator
apt-get install libpam0g-dev libqrencode3
cd ~
wget https://google-authenticator.googlecode.com/files/libpam-google-authenticator-1.0-source.tar.bz2
tar jvxf libpam-google-authenticator-1.0-source.tar.bz2
cd libpam-google-authenticator-1.0/
make
make install

Open a shell for a user you want to enable two-factor authentication for and run ‘google-authenticator’ to configure.

google-authenticator
Configure Google Authenticator
Configure Google Authenticator

Nice!

Configure your Mobile device
Install the ‘Google Authenticator’ app and just scan the QR-code. It will automatically configure itself and start displaying verification codes.

Verification codes displayed by app
Verification codes displayed by app

You should notice it’ll display new codes each 30 seconds.

Configure SSH
Two files need to be edited in order to enable two-factor authentication in SSH.

vim /etc/pam.d/sshd

Add this line:

auth required pam_google_authenticator.so nullok

Where to put this in the file? That depends. When you put it at the top, SSH will first ask a verification code, then a password. To me this sounds unlogical, so I placed it just below this line:

@include common-auth

The ‘nullok’ option, by the way, tells PAM whenever no config for 2-factor authentication is found, it should just ignore it. If you want SSH logins to fail, when no two-factor authentication is configured, you can delete the option. Be warned to at least config it for one user, or you will be locked out of your server.

Now tel SSH to ask for the verification code:

vim /etc/ssh/sshd_config

Edit the setting, it’s probably set to ‘no’:

ChallengeResponseAuthentication yes

Now all you need to do is restart SSH. Keep a spare SSH session logged-in, just in case.

/etc/init.d/ssh restart
SSH 2-factor authentication in action
SSH two-factor authentication in action

SSH will now ask for a verification code when you do an interactive login. When a certificate is used, no verification code is asked.

Configuring Ubuntu Desktop
The Ubuntu desktop can also ask for a verification code. Just edit one file:

vim /etc/pam.d/lightdm

And add this line, just like SSH:

auth required pam_google_authenticator.so nullok

The login screen should now ask for a verification code. It looks like:

Two-factor authentication in Ubuntu
Two-factor authentication in Ubuntu

You can setup any PAM service like this, the basic principles are all the same.

Skip two-factor authentication if logging in from the local network
At first this is all very cool, but soon it becomes a bit annoying, too. When I SSH from a local network, I just don’t want to enter the verification code because I trust my local network. When I SSH from remote, a verification code is required. One way to arrange that, is always login with certificates. But there is another way to configure it: using the pam_access module. Try this config:

auth [success=1 default=ignore] pam_access.so accessfile=/etc/security/access-local.conf
auth required pam_google_authenticator.so nullok

The config file, /etc/security/access-local.conf looks like:

# Two-factor can be skipped on local network
 + : ALL : 10.0.0.0/24
 + : ALL : LOCAL
 - : ALL : ALL

Local login attempts from 10.0.0.0/24 will not require two-factor authentication, while all others do.

Two-factor only for users of a certain group
You can even enable two-factor authentication only for users of a certain group. Say, you’d want only users in the ‘sudo’ group to do use two-factor authentication. You could use:

auth [success=1 default=ignore] pam_succeed_if.so user notingroup sudo
auth required pam_google_authenticator.so nullok

By the way, “[success=1 default=ignore]” in PAM means: on success, skip the next 1 auth provider (two-factor authentication), on failure, just pretend it didn’t happen.

Moving the config outside the user’s home
Some people prefer not to store the authentication data in the user’s home directory. You can do that using the ‘secret’ parameter:

mkdir -p /var/lib/google-authenticator/
mv ~remibergsma/.google_authenticator /var/lib/google-authenticator/remibergsma
chown root.root /var/lib/google-authenticator/*

The PAM recipe will then become:

auth required pam_google_authenticator.so nullok user=root secret=/var/lib/google-authenticator/${USER}

As you’ve seen, PAM is very flexible. You can write almost anything you want.

Try the Google Authenticator two-factor authentication. It’s free and adds another security layer to your home server.

When it not possible to reach a server you want to SSH to directly, you can make use of SSH’s built in capability to chain multiple commands. Suppose you have a network setup like in the image below.

Network overview
Network overview

Firewalls or ACL’s prevent direct access to the ‘web server’ in network #2. In between is a ‘jump host’ in network #1. A ‘jump host’ is a host you can SSH to, and from there reach the next hop. How to SSH to the web server?

You could do this manually:

ssh -l user jump-host

and then from that server:

ssh -l user webserver.dmz

But using the -t switch, you can chain them together like this:

ssh -A -t -l user jump-host \
ssh -A -t -l user webserver.dmz

The -A switch enables forwarding of the ssh-agent. When using key based authentication, you’ll be able to login with typing the certificate’s password only once.

Using this technique, you can also build a SSH tunnel through the jump host:

ssh -A -t -l user jump-host \
-L 8080:localhost:8080 \
ssh -A -t -l user webserver.dmz \
-L 8080:localhost:8080

When you type: http://localhost:8080 in a browser, you are connected over a secure tunnel to the web server in Network #2. Thanks to the chaining of commands, this is now possible.

You can use many chained commands, so this is very flexible.

Recently I was wondering if a Raspberry Pi would be able to run an accurate clock and provide a stable NTP service.

A virtual machine can’t do it, because its system clock has a changing drift factor. The clock compensation technique built into NTP was never designed to handle anything but a static drift factor. This is unfortunate because NTP is a light-wight protocol, so using physical hardware for it may sound wasteful. The Raspberry Pi is physical hardware with a very small footprint. So, how accurately can the Raspberry Pi keep time?

Let’s start with some background before we answer the question.

Clocks in Linux
Two clocks are important in Linux: a ‘hardware clock’, also known as RTC (Real Time Clock), CMOS or BIOS clock. This is the battery backed clock on the motherboard that keeps time even when the machine is shut down. The second clock is called the ‘system clock’, maintained by the operating system, Linux in our case. At boot time, the hardware clock is used to set the system clock and from that point onwards the system clock is used to track time. At shutdown, the system time is used to set the hardware clock as a way to ‘save’ the time between reboots. Linux uses a clocksource to maintain time, several of which exist, mostly depending on the available hardware.

Just like virtual hardware, the Raspberry Pi has no hardware clock. But it is fairly simple to add a hardware clock using the GPIO ports. In a previous blog post, I explained how to set this up.

This is a Raspberry Pi with a hardware clock
This is a Raspberry Pi with a hardware clock

Clock accuracy
Clock accuracy is defined in terms of ppm (parts per million) and it gives a way of comparing accuracies. The hardware clock I use, a RasClock, is said to lose 3 ppm (parts per million). So, 3 of each 1 million seconds may be wrong. This means:

3/10e6 x 24 x 60 x 60 = 0.2592 sec/day may be wrong.

One year after setting the hardware clock, it will be 95 seconds off at max.

Note: an atomic clock’s accuracy is something in the order of 0.0000001 ppm 😉

How Linux keeps time
Because the hardware clock is used only at system start, it doesn’t help in keeping time while the system runs. The OS needs a decent clocksource to be able to keep time. Think of the clocksource as a counter maintained by the kernel. The purpose of the clocksource is to provide a timeline for the system that tells you where you are in time. For example issuing the command ‘date’ on a Linux system will eventually read the clocksource to determine exactly what time it is.

On x86 and/or amd64 hardware, common clocksources are TSC (Time Stamp Counter), acpi_pm and since about 2005 HPET (High Precision Event Timer).

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource

The first command shows the current clocksource, the second all available ones.

Rasberry Pi’s clocksource
The Raspberry Pi does not ship with a TSC nor HPET counter to use as clocksource. Instead it relies on the STC that Raspbian presents as a clocksource. Based on the source code, “STC: a free running counter that increments at the rate of 1MHz”. This means it increments every microsecond.

How accurate is this STC?
Now that I have a hardware clock hooked to the Raspberry Pi, I can compare the STC clocksource (ie the system clock) to the hardware clock. First I made sure nothing is adjusting the clocksource already, or else you will not get accurate results. To double check:

vim /etc/default/adjtimex

Make sure these vars have their default values:

TICK=10000
FREQ=0

And restart afterwards:

/etc/init.d/adjtimex restart

These values are used to adjust the system clock. A value of FREQ=65536 speeds up the system clock by about 1 ppm, or .0864 sec/day. FREQ=6553600 and TICK=1 are equal and both speed up the system clock by about 100ppm, or 8.64 sec/day. Since we want to find out the precision of the Raspberry Pi’s system clock, we should not adjust anything. The system should not be doing other tasks during the test, other than normal OS operations.

First I synced the hardware clock with the system clock and logged the point in time that I did this:

hwclock --hctosys --utc; date -u

Read both the hardware clock and the system clock to verify:

hwclock -r; date -u

The two were equal in my test. To compare the hardware clock with the system clock in a more detail, a tool called ‘adjtimex’ exists. I used it with the –compare flag to run the test, like this:

adjtimex --compare=1 --utc

Warning: when you install ‘adjtimex’, it generates adjustment values in ‘/etc/default/adjtimex’. Make sure to reset them, like I did above or else you will get incorrect results.

Example output of the above command:

 --- current --- -- suggested --
cmos time system-cmos error_ppm tick freq tick freq
1367834856 -0.002202

Since both clocks just synced, there shouldn’t be much of a difference. I used the following command to log these values during a few days:

while :; do \
date -u +'# Generated at: %Y-%m-%d %H:%M:%S UTC %s %N'\
--date '+1 second'|
tee -a time.log 2>&1 \
&& adjtimex --compare=1 --utc \
| grep -v current | grep -v cmos |\
tee -a time.log 2>&1 \
&& sleep 1;\
done

This command enters an infinite loop and writes results to a logfile. Each loop takes about 2 seconds, so effectively this measures every 2 seconds the difference between the hardware clock and the system clock.

Afterwards, using some vim magic, I combined both logfiles, created a CSV from it and did most of the test analysis in Libre Office. Depending on the period you compare, you may want to filter out some records. Say you want a 1 minute interval in the rest results, instead of 2 seconds. I’ve found awk to be very useful at this:

awk 'NR == 1 || NR % 30 == 0' file.csv > file-1min-interval.csv

I’ve run the test command on two equally equipped and setup Raspberry Pi’s.

timedrift_raspberrypi_stc_versus_rasclock_rtc_full

You can see the drift is fairly static and predictable. I’ve measured every 2 seconds during roughly 59 hours.

timedrift_raspberrypi_stc_versus_rasclock_rtc_1hour

The above chart shows the first hour in more detail.

timedrift_difference_raspberrypi_1_and_2

There is a small difference between the two Raspberry Pi’s I tested, probably due to the fact the STC clocksource is entirely done in software (it does not use a hardware crystal of some kind). After 59 hours the difference is 0.242016 seconds in my test. More testing is needed to find out if this difference is consistent. It might even be caused by something else. Although I’m not sure, I wouldn’t worry too much about it

Detailed results:

Pi #1
Total deviation: 8.763317 sec
Accuracy: 8.763317 / ( 59 x 60 x 60 ) x 10e6 = 41.259ppm

Pi #2
Total deviation: 8.521301 sec
Accuracy: 8.521301 / ( 59 x 60 x 60 ) x 10e6 = 40.119ppm

Average accuracy: ( 41.259 + 40.119 ) /2 = 40.689ppm

Deviation in 24h: ( 40.689 / 10e6 ) x 24 x 60 x 60 = ~3.52 sec. So, without adjustments or syncs with other sources, it will drift this many seconds in 24 hours.

Conclusion
While the STC clocksource in Raspbian is not as accurate as a hardware clock, it does a great job in keeping time. The timedrift is linear and it has an accuracy of ~40ppm. It means the Raspberry Pi is able to keep time pretty well just on its own. This a requirement to build a solid NTP service and it is exactly what a VM cannot do. Therefore the Raspberry Pi is an interesting device to consider when planning your NTP service.

We now know what the Raspberry Pi does when it is not synced with other sources. Of course we’ll setup a system where it does sync to make sure it will drift away as little as possible. In a follow-up post we’ll continue from here in building a NTP service on the Raspberry Pi.

The Raspberry Pi does not have a hardware clock on board. Instead, to keep track of time during reboots and downtime the ‘fake-hwclock’ package is used. It contains scripts to save the kernel’s current clock periodically (including at shutdown) and restore it at boot so that the system clock keeps at least close to realtime. Combined with NTP this is a simple, cheap and fairly accurate setup for most use-cases.

For a project at work I wanted to test a Raspberry Pi with a Real Time Clock (RTC) connected through the GPIO pins. This hardware clock needs to be very precise. I found the RasClock, by Afterthought Software, to be precise enough (about 3ppm or ~95 seconds deviation per year) while still affordable (about 15 euro).

Hardware installation

The hardware installation is simple. Just insert the battery and place the RasClock on the last 6 GPIO ports, at the end of the board. That’s it.

This is how the RasClock GPIO module looks like
This is how the RasClock GPIO module looks like
This is the RasClock with a battery backup installed
This is the RasClock with a battery backup installed
This is the RasClock installed on the Raspberry Pi, side view
This is the RasClock installed on the Raspberry Pi, side view
This is the RasClock installed on the Raspberry Pi, top view
This is the RasClock installed on the Raspberry Pi, top view
This is the RasClock installed on my two boxed Raspberry Pi's
This is the RasClock installed on two boxed Raspberry Pi’s

Looks pretty good!

Software installation

I recommend upgrading to the latest Raspbian version. At the time of writing Raspbian runs on kernel 3.6.11. By far the easiest way to upgrade firmware, is to use rpi-update.

Unfortunately the ‘rtc-pcf2127a’ module needed to operate the RasClock, is not in the default Linux kernel provided by Raspbian. Afterthought Software provides both a binary packages and source code for their forked raspberrypi/linux repository. They added an I2C driver for NXP/Philips PCF2127A device from Eckelmann AG.

To install the binary package:


wget http://afterthoughtsoftware.com/files/linux-image-3.6.11-atsw-rtc_1.0_armhf.deb
dpkg -i linux-image-3.6.11-atsw-rtc_1.0_armhf.deb
cp /boot/vmlinuz-3.6.11-atsw-rtc+ /boot/kernel.img

The binary package is nice and quick for testing. Installing from source allows you to only build the needed kernel module, instead of a complete kernel replacement.

In the comments of the Afterthought Software site, someone posted these steps. Thanks for sharing your work!


sudo apt-get -y install build-essential gcc make cmake i2c-tools
mkdir devel
cd devel
wget https://github.com/raspberrypi/linux/archive/rpi-3.6.y.tar.gz
wget https://github.com/afterthoughtsoftware/linux/commit/fd5ff2d88f470ed70ff58393eee61110b181816a.patch
tar vxzf rpi-3.6.y.tar.gz
cd linux-rpi-3.6.y/
SUW=`pwd`
patch -p1 < ../fd5ff2d88f470ed70ff58393eee61110b181816a.patch
zcat /proc/config.gz > .config
sed -i 's/# CONFIG_RTC_DRV_PCF2127A is not set/CONFIG_RTC_DRV_PCF2127A=m/g' .config
echo m | make oldconfig
wget https://github.com/raspberrypi/firmware/raw/master/extra/Module.symvers
make modules_prepare
cd drivers/rtc
make -C $SUW M=`pwd`
sudo cp rtc-pcf2127a.ko /lib/modules/3.6.11+/kernel/drivers/rtc/
sudo depmod

It doesn’t matter if you install the binary package or compile the kernel module on your own. The steps below are the same either way.

Make sure to load the needed modules:


cat >> /etc/modules <<EOL
i2c-bcm2708
rtc-pcf2127a
EOL

Make sure the i2c-tools package is installed:


apt-get install i2c-tools

I talked about the ‘fake-hwclock’ package. Now that we have a hardware clock, we should remove this package and it’s crons.


apt-get remove fake-hwclock
rm /etc/cron.hourly/fake-hwclock
update-rc.d -f fake-hwclock remove
rm /etc/init.d/fake-hwclock

Enable the ‘hwclock.sh’ script (part of util-linux), instead:


update-rc.d hwclock.sh enable

It’s now time to boot the new kernel. When you reboot, the RasClock will be available as /dev/rtc0.


dmesg

The kernel ring buffer should list something like this:

[ 32.737903] rtc-pcf2127a 1-0051: chip found
[ 32.739712] rtc-pcf2127a 1-0051: rtc core: registered rtc-pcf2127a as rtc0
[ 32.739775] i2c i2c-1: new_device: Instantiated device pcf2127a at 0x51

Because this is a new device, we need to set the time in the hardware clock. Do this by copying the system time to the RasClock.


hwclock --systohc --utc

Using the ‘hwclock’ command, you can read the RasClock:


hwclock --show --utc

Output:

Wed 08 May 2013 21:59:49 CEST -0.862324 seconds

Adding the ‘–debug’ flag adds some more interesting output, especially if you want to know exactly what goes on. Example output:

hwclock from util-linux 2.20.1
Using /dev interface to clock.
Assuming hardware clock is kept in UTC time.
Waiting for clock tick...
/dev/rtc0 does not have interrupt functions. Waiting in loop for time from /dev/rtc0 to change
...got clock tick
Time read from Hardware Clock: 2013/05/08 19:56:20
Hw clock time : 2013/05/08 19:56:20 = 1368042980 seconds since 1969
Wed 08 May 2013 21:56:20 CEST -0.752934 seconds

You can also access the hardware clock through the /sys pseudo filesystem like this:


cat /sys/class/rtc/rtc0/date
cat /sys/class/rtc/rtc0/name

The first command returns the date, the second command the name of the chip which is ‘rtc-pcf2127a’.

Using the clock

In my current setup, the RasClock keeps time during reboot and when the Raspberry Pi is turned off. My goal is to build a NTP server that is capable of keeping time also when no NTP peers are available to prevent time drifts. I assume the hardware clock that I have now available is more accurate than the clocksource that drives the system time. I need to experiment to see how the two clocks compare. I’ll come back on the subject in a follow-up post.

When a SSH-key changed, this warning is displayed:

ssh -l username 172.16.12.34

Warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
aa:bb:cc:a2:b6:87:bd:43:f9:ff:<wbr />02:8e:a6:b8:29:42.
Please contact your system administrator.
Add correct host key in /home/remi/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/remi/.ssh/known_hosts:6

In case you know the host identification has changed, you can safely discard this warning.

You could run ‘vim ~/.ssh/known_hosts‘ enter ‘6G‘ to go to the 6th line, ‘dd‘ to delete that line and finally ‘:wq‘ to save the file. But, wouldn’t a one-liner be handy?

Try:

ssh-keygen -R 172.24.111.132

Output:

/home/remi/.ssh/known_hosts updated.
Original contents retained as /home/remi/.ssh/known_hosts.

I’ve tested this on Ubuntu, Debian, SUSE and Red Hat Enterprise Linux.

In case you get this error:

fopen: No such file or directory

There isn’t a ‘known_hosts’ file in ‘~/.ssh/’. You can use the -f flag to specify the right file.

At times it can be very handy to be able to use a local MySQL socket, while the real MySQL server is in fact a remote one. I once used this technique to split a dozen LAMP servers into dedicated (clustered) web servers and MySQL servers. Without having all clients to update scripts, that is.

Today I ran into another use-case: while migrating an old 32bit MySQL Windows server to a modern Linux VM. We wanted to get some statistics from it. The famous ‘mysqltuner.pl‘ script is designed to run from localhost. Well, let’s fake MySQL is on localhost then 😉

For these tricks to work you need a tool called ‘socat‘, which can be installed like this:

apt-get install socat

This is how to create a socket in ‘/var/lib/mysql/mysql.sock’, owned by both user and group ‘mysql’ and forward the connections to ‘mysql-server’ on port 3306:

socat UNIX-LISTEN:/var/lib/mysql/mysql.sock,fork,\
reuseaddr,unlink-early,user=mysql,group=mysql,mode=777 \
TCP:mysql-server:3306 &

You can now connect:

mysql -u user -p

No need to supply a hostname, MySQL connects to the local socket by default and socat takes care of forwarding the connection to the real MySQL server.

It is important to note this works as long as socat is running. You could use a tool called runit to keep socat running at all times. In another post I’ll explain more about runit.

This works with other protocols as well.