Remi Bergsma's blog

A few weeks ago I attended a full week of SUSE training, which I really enjoyed. There were lots of highly skilled geeks and the course had a nice solid pace. By the end of the week I did the CLE practicum exam (to become a Certified Linux Engineer). Today I heard I passed! 🙂

I’ve been working with SUSE Linux Enterprise Server since I started my new job back in December and the training really helped me.

Before I’m officially certified, there’s another step to take though. One needs to pass three exams: CLA, CLP and CLE. I’ve passed both CLA and CLE now, so I only need to pass the one in between. This happened because the course I attended was aimed towards RHCE certified professionals, which I am not (yet). So, either passing the CLP or RHCE practicum will bring in the CLE certification as well. Looking forward to continue the certification process!

PS: See this post for an overview of Linux certifications.

When you create firewall rules with iptables on Linux, you want to make them persistent over reboot, because they are not by default. Different Linux distributions have different methods of achieving this, although the basics are similar. I’ve been working with Debian, Red Hat Enterprise Linux and SUSE Linux Enterprise Server and in this blog I’ll describe how to configure each of them to save your iptables rules across reboots.

First the good news: the iptables package, the administration tool for packet filtering and NAT, always ships with Linux distributions. The package also includes the ‘iptables-save‘ and ‘iptables-restore‘ tools. These do what you might already expect from their names: save or restore iptables rules. ‘iptables-save‘ outputs to stdout, which you can save to a file:

iptables-save > /etc/iptables/rules

To load these again:

iptables-restore < /etc/iptables/rules

These really are the basics that work across Linux distributions and that you can use in your custom boot scripts. In addition to this, each Linux distribution has its own way to make this process easier.

Red Hat Enterprise Linux (RHEL):

RHEL (and the same counts for CentOS and Fedora) has some built in mechanism to help automate this. First of all, there are some settings in ‘/etc/sysconfig/iptables-config‘:

IPTABLES_SAVE_ON_STOP=”yes”
IPTABLES_SAVE_ON_RESTART=”yes”

You can set them to “yes” to have persistent iptables rules. In fact, there are many more settings in that file that allow for finer control. That’s all, since the rest is handled automatically.

At any time it’s possible to save the current state. Just run:

service iptables save

And it will, like on reboot, save the rules to: ‘/etc/sysconfig/iptables‘. Pretty easy and pretty powerful!

SUSE Linux Enterprise Server (SLES 11):

SLES (and the same counts for OpenSUSE) is yet another story. SLES 11 now ships with SUSE Firewall. Instead of defining the rules yourself, you tell Yast what you want to achieve and it generates the needed iptables rules for you. Although SUSE Firewall does allow you to add custom rules, it isn’t really designed for it. The tool is pretty nice though, because it integrates fully with Yast and allows for easy maintenance of rules. When you install a package, it automatically opens the associated port, for example.

This all might seems a bit scary for us sysadmins, right?! Don’t worry, it’s still possible to manage rules on your own by disabling SUSE Firewall. But have a look at it first, as you might as well like it.

To start the SUSE Firewall admin module, run:

yast2 firewall

The interface is pretty self-explaining. Afterwards, to activate the changes run:

SuSEfirewall2

It’s even possible to by-pass Yast, and edit the config file directly. It’s safe to combine the two methods, no problem.

vim /etc/sysconfig/SuSEfirewall2

For example, to open a port you’d edit the ‘FW_SERVICES_EXT_TCP’ variable. Just make a list (space separated) with protocols you want to allow. These protocols refer to files in ‘/etc/sysconfig/SuSEfirewall2.d’.

Like with using Yast, activate the changes when you’re done.

SuSEfirewall2

I’ve used it for some time and it’s actually pretty easy. It just depends on the project whether or not to use it, I guess.

Debian
When I was using Debian (same counts for Ubuntu as well), I used to create a small shell script and place it in ‘/etc/network/if-pre-up.d’. Just before the network interface is brought up, the iptables rules will be restored. The idea is to do the same when the interface goes down (use the ‘/etc/network/if-post-down.d’ folder to place the script in). Using these thechniques, you can create something and have fine control over it.

Recently I heard about a tool called ‘iptables-persistent’ that can automate this out-of-the-box. Here’s the package description:

iptables-persistent - boot-time loader for iptables rules: Current iptables rules can be saved to the configuration file '/etc/iptables/rules.v4'. These rules will then be loaded automatically during system startup.

To install it:

sudo apt-get install iptables-persistent

During install, the program asks to save both ipv4 and ipv6 iptables rules. Please note this counts for Wheezy, the current stable release uses the file ‘/etc/iptables/rules’.

To manually save the iptables rules, run:

/etc/init.d/iptables-persistent save

Although this should be done automatically when you reboot. It looks like the Red Hat way of doing things, but just with an extra package installed.

Conclusion: 
iptables all over the place, just with different tooling to automate it 🙂

The Raspberry Pi can easily be setup as an OpenVPN server. One common feature is to access servers or services on the remote network. Another use case is to provide a secure connection when you’re not at home.

You can do this by sending all traffic over the VPN connection, instead of only traffic for the remote servers. The idea is to connect your laptop to your OpenVPN server (this is encrypted by default) and access the internet from there. Whatever you send over the wifi connection, is encrypted by the VPN and thus safe. In the comments on the previous post there was someone trying to set this up. To help him solve the issue, I tested this myself, and decided to write a post showing how to do it because I really like the idea.

This is how it looks like: you’re on a untrusted network (red) and create a safe VPN connection (green) and all traffic will flow over the green network, to the also trusted home network (lighter green). People on the red network now cannot see the sites you visit.

Here’s how to set it up:

1. Allow the OpenVPN server to route ip traffic
2. Tell the default gw where to send traffic for OpenVPN clients
3. Send DNS servers to the VPN Clients
4. Configure the OpenVPN Client
5. Test your setup

Allow the OpenVPN server to route ip traffic
First of all, you need to allow the OpenVPN server to route packets.

sudo vim /etc/sysctl.conf

Add this line, or alter it if it already exists:

net.ipv4.ip_forward = 1

Then activate the change:

sudo sysctl -p

The change is now activated, and persistent.

Tell the default gw where to send traffic for OpenVPN clients
Another important step is to tell the default gateway in the home network (lighter green) where to send traffic for theOpenVPN clients. If you omit this step, this traffic gets lost. More info in this post. Short story: add this static route to your default gateway. If it’s Linux, you’d run:

sudo route add -net 10.8.0.0/24 gw 10.5.5.5

Assuming 10.5.5.5 is the ip address of the OpenVPN server. When your router is not Linux, check the manual on howto add a static route.

Send DNS servers to the VPN clients
Most ISP’s restrict the usage of their DNS servers to their own network. When you connect to Wifi, you probably receive some DNS servers via DHCP. When you connect to VPN and then send all traffic through the VPN, you are effectively using the network (and internet connection) of your VPN server and not the local network. The DNS-servers you received via DHCP might not work because you access them from another network. To solve this, configure the OpenVPN server to push public DNS servers to use. Alternatively you can also push some local DNS servers or the DNS servers of your ISP.

sudo vim /etc/openvpn/server.conf

Add or edit these lines:

push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"

These are Google’s public DNS servers. Just enter some DNS servers that work on the network of your OpenVPN server and that you are allowed to use.

Configure the OpenVPN client
I’m using Viscosity and all I have to do is enable a setting to send all traffic over the VPN connection.

Alternative way: It’s also possible to configure the ‘Send all traffic over VPN connection’ on the server-side instead of the client. Both has pros and cons, of course. To set it server-side set this option in the server config:

push "redirect-gateway def1"

Test your setup
The final step: test your setup! Make sure you’re connected through some other network than the one the OpenVPN server is in. Use some public Wifi service of connect over 3G. My iPhone can share its 3G connection and start a Wifi hotspot. My MacBook connects via Wifi, then does the OpenVPN connection to the Pi. The expected behavior is then to see the public ip address of the OpenVPN server’s internet connection, instead of the 3G ip address. Use a website like whatismyip.com to test this. Do this before connecting the VPN, and after. It should be different.

Enjoy your secure browsing experience!

I’m using SSH’s key-based authentication to login to my servers using a SSH key pair. Of course my private key is protected with a password of its own. This has the consequence that even with key-based authentication, you still need to type a password: your private key passphrase. And you need to type it on every single connect. Although this is far more secure that having no passphrase on your private key, typing it so many times is a bit over done.

In this blog I’ll show you how to setup a system where you enter the passphrase only once per day when you start a new terminal session. For me this works great, as I now only type my passphrase in the morning when I login to my workstation and am able to connect password-less the rest of the day. Whenever I leave my desk, my screen is locked so no one can access my key.

Setting up key-based authentication

The first step is to generate a key pair to use.

ssh-keygen -t rsa -b 2048

The output is something like below. Enter your passphrase twice, when prompted.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/remi/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/remi/.ssh/id_rsa.
Your public key has been saved in /home/remi/.ssh/id_rsa.pub.
The key fingerprint is:
41:22:af:8b:4b:aa:94:34:a4:fa:5a:eb:91:4c:5b:5e remi@wks
The key’s randomart image is:
+–[ RSA 2048]—-+
|    . . .        |
|     o o         |
| .    . .        |
|o    .   .       |
|.o. o E S        |
|ooo* o           |
|.oO o            |
|.= +             |
|=o=              |
+—————–+

ssh-copy-id -i ~/.ssh/id_rsa.pub user@server

export SSH_AUTH_SOCK=$(find /tmp/ssh-* -user `whoami` -name agent\* -printf '%T@ %p\n' 2>/dev/null | sort -k 1nr | sed 's/^[^ ]* //' | head -n 1)

#!/bin/bash
echo “Updating ssh-agent socket environment..”
echo “Current value:  $SSH_AUTH_SOCK”
export SSH_AUTH_SOCK=$(find /tmp/ssh-* -user whoami -name agent* -printf ‘%T@ %p\n’ 2>/dev/null | sort -k 1nr | sed ’s/^[^ ]* //’ | head -n 1)
if [ $? -gt 0 ]; then
echo “ERROR!”
exit $?
fi
echo “New value:  $SSH_AUTH_SOCK”
echo “All done!"

. ~/update-ssh-agent-socket.sh

About 2 weeks ago I wrote in a previous post about LPIC, when I had just passed the LPIC-1 exams. In the mean time I studied the objectives, read some man-pages and experimented with the things I wasn’t 100% familier with.

And today I completed LPIC-2! 🙂 There were two exams required to pass. I scored 740/800 on the 201 exam, and 730/800 on the 202 exam.

Now I’m looking forward on taking some SUSE classes and exams in a couple of weeks!

OpenVPN is a very powerful tool to connect to a remote network in a secure and easy way. But while it’s quite easy to set it up (see my post about OpenVPN on Raspberry Pi), getting all the pieces together takes some more network understanding and configuration. I created an image to show what’s going on when a OpenVPN client connects to the server.

The grey network is the local network you’re connecting to, the green network is created by OpenVPN. After a successful connection, the OpenVPN server can ‘push’ a route to the OpenVPN client to make it aware of the grey network that is available through the OpenVPN connection. In this case you’d add this setting to the OpenVPN server:

push "route 10.5.5.0 255.255.255.0"

Now that the OpenVPN client knows how to find the Linux server on the grey network, it should be working, right? Nope. The Linux server does not know about the green network. So, when a OpenVPN client connects with a 10.8.0.x ip-address, this server does not know where to send the response to. When this happens, the Linux server sends its traffic to the default gateway of the local network: the router of the grey network.

But unfortunately, the router does not know about the green network neither. Since 10.8.0.x is a non-routable address (RFC 1918), it drops the packets because it has no way of knowing where to find this private network. To the OpenVPN client, this may look like there’s something wrong with the VPN connection, because even a simple ping will not work and times out. In reality, this is a routing problem.

The solution? Make the Linux server or at least the router aware of the green network, and tell them where to send traffic to this network. This is done by adding a route like this:

route add -net 10.8.0.0/24 gw 10.5.5.5

Assuming the local network ip-address of the OpenVPN server is 10.5.5.5.

This way the hosts in the grey network know about the green network and send their traffic to the OpenVPN server. Of course the OpenVPN server knows how to reach the OpenVPN client and all is working!

You can extend this further and add routes to the network of the VPN client and make hosts on both networks communicate to each other. This is useful for example to connect a branch-office to the corporate network. I’ll write about this in more detail some other time.

Hopefully this post brings some insight in what goes on when using networks and VPN connections.

Someone asked me if it were possible to download a web site and make it available offline. To some extend, this can be done. Interactive forms will not work (searching, ordering, etc), but you can use ‘wget‘ to transform a website into a static version.

It goes like this:

wget \
 --recursive \
 --no-clobber \
 --page-requisites \
 --html-extension \
 --convert-links \
 --restrict-file-names=windows \
 --domains example.org \
 --no-parent \
 --wait=1 \
 --limit-rate=500K \
 example.org/

Let me explain:
The ‘–recursive’ option downloads the entire web site and ‘–domains’ tells wget not to follow links outside example.org. Otherwise you will download far too many pages. ‘–page-requisites’ makes sure we’ll get all the elements that compose the page (images, CSS, etc), ‘–html-extension’ saves files with the .html extension so they will work on a stand-alone pc, ‘–convert-links’ converts links so they’ll work off-line and ‘–no-clobber’ prevents any existing files to be overwritten

Using a ‘–limit-rate’ you can prevent wget from using all available bandwidth. Wile downloading will take longer, it is now possible to browse the web while wget is downloading.

Give it a try, it works pretty nice and is great if you’re about to make big changes to your site and you want to save a copy of the old version.

The Raspberry Pi is a $35 credit-card sized computer, with an ARM-based CPU. It uses very little power (only 3 Watt), so it’s ideal for a server that’s always-on. I was thinking what’d be a nice task for my Raspberry Pi and came up with an OpenVPN server. This enables me to connect to my home from anywhere, for example to access some files or to access the internet from there.

Before we start, Let’s have a look what’s on board the Raspberry Pi so you’ve an idea what we talk about:

Here’s mine in action:

Now, let’s see how we can turn it into a OpenVPN server. Actually this is very easy due to the fact that the Raspberry Pi is running (a modified) Debian Weezy called Raspbian. Since it’s Debian, you can use apt-get to install software:

apt-get install openvpn

After the install finishes, you need to generate keys for the server and the client(s). OpenVPN ships with the ‘easy-rsa’-tool. It’s easiest to copy the example folder and work from there.

cp -R /usr/share/doc/openvpn/examples/easy-rsa /etc/openvpn
cd /etc/openvpn/easy-rsa/2.0

The ‘easy-rsa’-tool has a file called ‘vars’ that you can edit to set some defaults. That will save you time later on but it’s not required to do so.

Load the vars like this (note the two dots):

. ./vars

Then we need to generate keys:

./clean-all
./build-ca
./build-key-server server
./build-key client-name
./build-dh

The first line makes sure we start from scratch. The second generates a key for the Certificate Authority. The key for the server itself is generated on the third line. Repeat the forth line for each client that needs to connect. Finally, we need the Diffie Hellman key as well, which is generated on the fifth line. Make sure you use a 2048 bit key, as suggested in the comments.

We need to copy the keys to the OpenVPN folder.

cd /etc/openvpn/easy-rsa/2.0/keys
cp ca.crt ca.key dh2048.pem server.crt server.key /etc/openvpn

Last step is to configure the server. You can copy the example config and make sure it points to the certs you just created.

cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn
gunzip /etc/openvpn/server.conf.gz
vim /etc/openvpn/server.conf

When you’re done, start OpenVPN like this:

/etc/init.d/openvpn start

Response looks like:

[ ok ] Starting virtual private network daemon: server.

Verify it by running:

ifconfig tun0

You’ll see:

tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 
 inet addr:10.8.0.1 P-t-P:10.8.0.2 Mask:255.255.255.255
 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
 RX packets:49 errors:0 dropped:0 overruns:0 frame:0
 TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:100 
 RX bytes:3772 (3.6 KiB) TX bytes:1212 (1.1 KiB)

Now you should be able to connect to the OpenVPN server with a client. I’m using Viscosity on Mac OSX, but there’re many clients available on almost any platform (Windows, Mac OSX, Linux). You need the client.crt, client.key and ca.crt files plus the ip-address of your Raspberry Pi.

Connecting should now work without trouble. Have a look at ‘/var/log/syslog’ to access the logfiles. You’d be able to see which client connects:

Jan 5 22:07:56 raspberrypi ovpn-server[14459]: 1.2.3.4:64805 [client-name] Peer Connection Initiated with [AF_INET]1.2.3.4:64805

Now that all is working, time for a last tip: when you want to access the network behind the Raspberry Pi through your OpenVPN connection, configure OpenVPN to push the right route to the clients. Edit the OpenVPN server config, and add a parameter like this:

push "route 10.1.7.0 255.255.255.0"

Be sure to enter the network- and netmask address that match your network setup. The route is automatically added on connect, and removed on disconnect.

Finally, enable routing on the Rasperry Pi:

echo 1 > /proc/sys/net/ipv4/conf/all/forwarding

Have fun with it, you can do a lot of great things with this little machine!

Update: Also see these follow-up posts that contain more detailed info on some interesting use cases and help you set it up:

HOWTO connect to hosts on a remote network using OpenVPN and some routing

Secure browsing via untrusted wifi networks using OpenVPN and the Raspberry Pi

The WordPress.com stats helper monkeys surprised me today with their 2012 annual report.

Here’s an excerpt:

My blog had 33,000 views in 2012. The busiest day of the year was September 4th with 644 views. The most popular post that day was How I replaced the CloudStack Virtual Router with my own physical Linux router that was featured on Linux Today.

Visitors came from 165 different countries.. wow! Most came from The United States, followed by The Netherlands and Germany. A lot of the visitors were searching on Google, some on Twitter, and found my blog that way.

Click here to see the complete report.

Thank you all and I wish you all the best in 2013!

See ya there 🙂

After passing my Bachelor of Computer Science I never took the time to take any specific Linux exam. But I’ve been working with Linux for about 14 years, so I know a lot about it. Now that I started my new job this month, I thought it was also time to get Linux certified.

I’ve been given the opportunity to get some training from SUSE (which is the distribution we use at work), and I’m really happy about that. Yes, I do have a nice employer 🙂

Let me tell you a bit about the different certification options for Linux. LPI is vendor neutral, whereas Red Hat and SUSE focus on their specific distributions of Linux. Still Linux, so the underlying knowledge is the same. The LPI Netherlands web site has a nice overview of all Linux certifications available (thanks!) that I’ve copied below:

One has to take the exams in the right order: start at the so called ‘Junior’-level, and work your way down. That’s why I decided to do some self-study and take the LPI-101 and LPI-102 exams to become LPIC-1 certified. I passed the first exam last week (730/800), and the second one today (700/800). As a bonus, I also received the Novell/SUSE Certified Linux Administrator (CLA) certificate. So this is a nice first step.

I’ll be taking the SUSE Certified Linux Professional (CLP) and SUSE Certified Linux Engineer (CLE) exams next month. Before that, I plan on doing some more self-study and take the LPIC-2 exams as well.

From what I’ve seen so far the LPI exams are nice and fill up some blanks in my knowledge. Especially LPIC-1 wasn’t too hard and I had fun going through the objectives. In fact, I already started looking at the LPI-201 exam and am looking forward to it. I come across most objectives in my day-to-day work; some more often than others of course. It’s nice to learn a new command or parameter and be able to use it from then on to do your job better. That is going forward!

I’ll keep you posted..