Archives For howto

When it not possible to reach a server you want to SSH to directly, you can make use of SSH’s built in capability to chain multiple commands. Suppose you have a network setup like in the image below.

Network overview

Network overview

Firewalls or ACL’s prevent direct access to the ‘web server’ in network #2. In between is a ‘jump host’ in network #1. A ‘jump host’ is a host you can SSH to, and from there reach the next hop. How to SSH to the web server?

You could do this manually:

ssh -l user jump-host

and then from that server:

ssh -l user webserver.dmz

But using the -t switch, you can chain them together like this:

ssh -A -t -l user jump-host \
ssh -A -t -l user webserver.dmz

The -A switch enables forwarding of the ssh-agent. When using key based authentication, you’ll be able to login with typing the certificate’s password only once.

Using this technique, you can also build a SSH tunnel through the jump host:

ssh -A -t -l user jump-host \
-L 8080:localhost:8080 \
ssh -A -t -l user webserver.dmz \
-L 8080:localhost:8080

When you type: http://localhost:8080 in a browser, you are connected over a secure tunnel to the web server in Network #2. Thanks to the chaining of commands, this is now possible.

You can use many chained commands, so this is very flexible.

The Raspberry Pi does not have a hardware clock on board. Instead, to keep track of time during reboots and downtime the ‘fake-hwclock’ package is used. It contains scripts to save the kernel’s current clock periodically (including at shutdown) and restore it at boot so that the system clock keeps at least close to realtime. Combined with NTP this is a simple, cheap and fairly accurate setup for most use-cases.

For a project at work I wanted to test a Raspberry Pi with a Real Time Clock (RTC) connected through the GPIO pins. This hardware clock needs to be very precise. I found the RasClock, by Afterthought Software, to be precise enough (about 3ppm or ~95 seconds deviation per year) while still affordable (about 15 euro).

Hardware installation

The hardware installation is simple. Just insert the battery and place the RasClock on the last 6 GPIO ports, at the end of the board. That’s it.

This is how the RasClock GPIO module looks like

This is how the RasClock GPIO module looks like

This is the RasClock with a battery backup installed

This is the RasClock with a battery backup installed

This is the RasClock installed on the Raspberry Pi, side view

This is the RasClock installed on the Raspberry Pi, side view

This is the RasClock installed on the Raspberry Pi, top view

This is the RasClock installed on the Raspberry Pi, top view

This is the RasClock installed on my two boxed Raspberry Pi's

This is the RasClock installed on two boxed Raspberry Pi’s

Looks pretty good!

Software installation

I recommend upgrading to the latest Raspbian version. At the time of writing Raspbian runs on kernel 3.6.11. By far the easiest way to upgrade firmware, is to use rpi-update.

Unfortunately the ‘rtc-pcf2127a’ module needed to operate the RasClock, is not in the default Linux kernel provided by Raspbian. Afterthought Software provides both a binary packages and source code for their forked raspberrypi/linux repository. They added an I2C driver for NXP/Philips PCF2127A device from Eckelmann AG.

To install the binary package:

wget http://afterthoughtsoftware.com/files/linux-image-3.6.11-atsw-rtc_1.0_armhf.deb
dpkg -i linux-image-3.6.11-atsw-rtc_1.0_armhf.deb
cp /boot/vmlinuz-3.6.11-atsw-rtc+ /boot/kernel.img

The binary package is nice and quick for testing. Installing from source allows you to only build the needed kernel module, instead of a complete kernel replacement.

In the comments of the Afterthought Software site, someone posted these steps. Thanks for sharing your work!

sudo apt-get -y install build-essential gcc make cmake i2c-tools
mkdir devel
cd devel
wget https://github.com/raspberrypi/linux/archive/rpi-3.6.y.tar.gz
wget https://github.com/afterthoughtsoftware/linux/commit/fd5ff2d88f470ed70ff58393eee61110b181816a.patch
tar vxzf rpi-3.6.y.tar.gz
cd linux-rpi-3.6.y/
SUW=`pwd`
patch -p1 < ../fd5ff2d88f470ed70ff58393eee61110b181816a.patch
zcat /proc/config.gz > .config
sed -i 's/# CONFIG_RTC_DRV_PCF2127A is not set/CONFIG_RTC_DRV_PCF2127A=m/g' .config
echo m | make oldconfig
wget https://github.com/raspberrypi/firmware/raw/master/extra/Module.symvers
make modules_prepare
cd drivers/rtc
make -C $SUW M=`pwd`
sudo cp rtc-pcf2127a.ko /lib/modules/3.6.11+/kernel/drivers/rtc/
sudo depmod

It doesn’t matter if you install the binary package or compile the kernel module on your own. The steps below are the same either way.

Make sure to load the needed modules:

cat >> /etc/modules <<EOL
i2c-bcm2708
rtc-pcf2127a
EOL

Make sure the i2c-tools package is installed:

apt-get install i2c-tools

I talked about the ‘fake-hwclock’ package. Now that we have a hardware clock, we should remove this package and it’s crons.

apt-get remove fake-hwclock
rm /etc/cron.hourly/fake-hwclock
update-rc.d -f fake-hwclock remove
rm /etc/init.d/fake-hwclock

Enable the ‘hwclock.sh’ script (part of util-linux), instead:

update-rc.d hwclock.sh enable

It’s now time to boot the new kernel. When you reboot, the RasClock will be available as /dev/rtc0.

dmesg

The kernel ring buffer should list something like this:

[ 32.737903] rtc-pcf2127a 1-0051: chip found
[ 32.739712] rtc-pcf2127a 1-0051: rtc core: registered rtc-pcf2127a as rtc0
[ 32.739775] i2c i2c-1: new_device: Instantiated device pcf2127a at 0x51

Because this is a new device, we need to set the time in the hardware clock. Do this by copying the system time to the RasClock.

hwclock --systohc --utc

Using the ‘hwclock’ command, you can read the RasClock:

hwclock --show --utc

Output:

Wed 08 May 2013 21:59:49 CEST -0.862324 seconds

Adding the ‘–debug’ flag adds some more interesting output, especially if you want to know exactly what goes on. Example output:

hwclock from util-linux 2.20.1
Using /dev interface to clock.
Assuming hardware clock is kept in UTC time.
Waiting for clock tick...
/dev/rtc0 does not have interrupt functions. Waiting in loop for time from /dev/rtc0 to change
...got clock tick
Time read from Hardware Clock: 2013/05/08 19:56:20
Hw clock time : 2013/05/08 19:56:20 = 1368042980 seconds since 1969
Wed 08 May 2013 21:56:20 CEST -0.752934 seconds

You can also access the hardware clock through the /sys pseudo filesystem like this:

cat /sys/class/rtc/rtc0/date
cat /sys/class/rtc/rtc0/name

The first command returns the date, the second command the name of the chip which is ‘rtc-pcf2127a’.

Using the clock

In my current setup, the RasClock keeps time during reboot and when the Raspberry Pi is turned off. My goal is to build a NTP server that is capable of keeping time also when no NTP peers are available to prevent time drifts. I assume the hardware clock that I have now available is more accurate than the clocksource that drives the system time. I need to experiment to see how the two clocks compare. I’ll come back on the subject in a follow-up post.

When a SSH-key changed, this warning is displayed:

ssh -l username 172.16.12.34

Warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
aa:bb:cc:a2:b6:87:bd:43:f9:ff:<wbr />02:8e:a6:b8:29:42.
Please contact your system administrator.
Add correct host key in /home/remi/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/remi/.ssh/known_hosts:6

In case you know the host identification has changed, you can safely discard this warning.

You could run ‘vim ~/.ssh/known_hosts‘ enter ‘6G‘ to go to the 6th line, ‘dd‘ to delete that line and finally ‘:wq‘ to save the file. But, wouldn’t a one-liner be handy?

Try:

ssh-keygen -R 172.24.111.132

Output:

/home/remi/.ssh/known_hosts updated.
Original contents retained as /home/remi/.ssh/known_hosts.

I’ve tested this on Ubuntu, Debian, SUSE and Red Hat Enterprise Linux.

In case you get this error:

fopen: No such file or directory

There isn’t a ‘known_hosts’ file in ‘~/.ssh/’. You can use the -f flag to specify the right file.

At times it can be very handy to be able to use a local MySQL socket, while the real MySQL server is in fact a remote one. I once used this technique to split a dozen LAMP servers into dedicated (clustered) web servers and MySQL servers. Without having all clients to update scripts, that is.

Today I ran into another use-case: while migrating an old 32bit MySQL Windows server to a modern Linux VM. We wanted to get some statistics from it. The famous ‘mysqltuner.pl‘ script is designed to run from localhost. Well, let’s fake MySQL is on localhost then 😉

For these tricks to work you need a tool called ‘socat‘, which can be installed like this:

apt-get install socat

This is how to create a socket in ‘/var/lib/mysql/mysql.sock’, owned by both user and group ‘mysql’ and forward the connections to ‘mysql-server’ on port 3306:

socat UNIX-LISTEN:/var/lib/mysql/mysql.sock,fork,\
reuseaddr,unlink-early,user=mysql,group=mysql,mode=777 \
TCP:mysql-server:3306 &

You can now connect:

mysql -u user -p

No need to supply a hostname, MySQL connects to the local socket by default and socat takes care of forwarding the connection to the real MySQL server.

It is important to note this works as long as socat is running. You could use a tool called runit to keep socat running at all times. In another post I’ll explain more about runit.

This works with other protocols as well.

When you boot your Raspberry Pi with Raspbian, it will get its ip address from a DHCP server. Most tutorials I’ve seen, ask you to login to a ‘router’ (that serves as DHCP server as well) to find the assigned ip address. When you haven’t access to the DHCP server, how do you find the Pi’s ip address?
You should know the MAC address of a Raspberry Pi always has a ‘b8:27:eb’ prefix. Using a Linux utility called ‘arp-scan‘ you can discover MAC addresses and their corresponding ip addresses. Arp-scan sends ARP packets to hosts on the local network and displays any responses that are received.
First, install arp-scan:
apt-get install arp-scan
Then locate all Raspberry Pi’s on the local network using this command:
arp-scan --interface=eth0 --localnet | grep b8:27:eb
You will need to be root, or arp-scan must be SUID root, in order to run arp-scan, because the functions that it uses to read and write  packets require root privilege.
Result:
172.16.12.34 b8:27:eb:00:11:ab (Unknown)
Using the ip address, it is now possible to SSH into your Raspberry Pi:
ssh -l pi 172.16.12.34

Today I figured out how to automatically add new devices (in my case those are mostly virtual machines) to the Zenoss monitoring system. This used to be done by hand, but no more 🙂

To add a new device (for example a Linux server called server001), simply call:

curl -u apiUser:apiPass \
'http://zenoss-server:8080/zport/dmd/DeviceLoader? \
deviceName=server001&devicePath=/Server/Linux&\
loadDevice:method=1'

It’s wise to create a dedicated Zenoss user just for these API calls, but you may use any account that has sufficient permissions to perform the action you’re calling.

As an alternative you can also use the ‘zenbatchload‘ command. Although you can only add new devices, not edit existing ones.The RESTful API does have the possibility to edit an existing device.

Let’s set some properties to the server we’ve just added:

curl -u apiUser:apiPass \
'http://zenoss-server:8080/zport/dmd/Devices/Server/\
Linux/devices/server001/manage_editDevice?serialNumber=1234&\
tag=tagname&productionState=1000&\
groupPaths=Group1&groupPaths=Group2&priority=3&\
comments=Api%20Test%20Comment&systemPaths=/System\
&rackSlot=Virtual'

It took me some time to figure out all available attributes, although once I found some of them I was able to Google the full list with an explaination.

Attributes:

  • deviceName – the name or IP of the device. If its a name it must resolve in DNS
  • devicePath – the device class where the first “/” starts at “/Devices” like “/Server/Linux” the default is “/Discovered”
  • tag – the tag of the [email protected]
  • serialNumber – the serial number of the device
  • zSnmpCommunity – snmp community to use during auto-discovery if none is given the list zSnmpCommunities will be used
  • zSnmpPort – snmp port to use default is 161
  • zSnmpVer – snmp version to use default v1 other valid values are v2
  • rackSlot – the rack slot of the device.
  • productionState – production state of the device default is 1000 (Production)
  • comments – any comments about the device
  • hwManufacturer – hardware manufacturer this must exist in the database before the device is added
  • hwProductName – hardware product this must exist in the manufacturer object specified
  • osManufacturer – OS manufacturer this must exist in the database before the device is added
  • osProductName – OS product this must exist in the manufacturer object specified
  • locationPath – path to the location of this device like “/Building/Floor” must exist before device is added
  • groupPaths – list of groups for this device multiple groups can be specified by repeating the attribute in the url
  • systemPaths – list of systems for this device multiple groups can be specified by repeating the attribute in the url
  • statusMonitors – list of status monitors (zenping) for this device default is “localhost”
  • performanceMonitor – performance monitor to use default is “localhost”
  • discoverProto – discovery protocol default is “snmp” other possible value is “none”

Finally, in case you wish to delete a device, that can be done as well:

curl -u apiUser:apiPass \
'http://zenoss-server:8080/zport/dmd/Devices/Server/Linux/\
devices/server001/deleteDevice'

Personally I prefer not to delete devices. I rather set the ‘productionState’ to ‘-1’ (Decommissioned) to keep the history in Zenoss.

These simple API calls make it possible to automatically add a new server to the monitoring, or sync information from another source. But you can use the API for gathering all sorts of data as well. For example the load-average:

curl -u apiUser:apiPass \
'http://zenoss-server:8080/zport/dmd/Devices/Server/Linux/\
devices/server001/getRRDValue?dsname=laLoadInt5_laLoadInt5'

Result:

2.0

If you want to start playing with it, have a look at the Zenoss API documentation.

Recently I was looking for a way to SSH from a network that blocked my outgoing SSH connection. I’d be nice to have a way around firewalls and be able to access your private Linux terminal. To be able to debug a problem from an remote location, for example.

A collegue suggested a tool called ‘Shell In A Box‘. Shell In A Box implements a web server that can export arbitrary command line tools to a web based terminal emulator using just JavaScript and CSS without any additional browser plugins. This means: connecting your browser via HTTPS to your own hosted Shell In A Box web site, and access a Linux terminal from there.

How cool is that? In this blog I’ll show you how to set it up in a secure way.

Building and installing Shell In A Box
I want to setup Shell In A Box on my Raspberry Pi. It’s a great device running Linux that has a very small energy consumption footprint. Ideal for an always-on device I’d say!

Since there is no package available, we’ve to compile our own. It’s best to get the sources from Github (original here), since the Github repository contains some patches and fixes for issues on Firefox.

These commands install the required dependencies, clone the Git repository and start building:

apt-get install git dpkg-dev debhelper autotools-dev libssl-dev libpam0g-dev zlib1g-dev libssl1.0.0 libpam0g openssl
git clone https://github.com/pythonanywhere/shellinabox_fork
cd shellinabox_fork
dpkg-buildpackage

During my first attempt, I ran into this problem:

dpkg-source -b shellinabox-2.14
dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../shellinabox_2.14.orig.tar.{bz2,gz,lzma,xz}
dpkg-buildpackage: error: dpkg-source -b shellinabox-2.14 gave error exit status 255

When grepping for ‘quilt’ I found a file called ‘/debian/source/format’. From what I can tell this does not do anything important, so I ended up deleting the file. Guess what, it now works.

rm ./debian/source/format

Build the package again, this should now succeed.

dpkg-buildpackage

This process will take some time (especially on the Raspberry Pi). Afterwards you’ll find the .deb file ready to be installed.

dpkg -i ../shellinabox_2.14-1_armhf.deb

I changed the configuration, to disallow the build-in SSL and to bind to localhost only. I did this because another web server will serve our terminal. I will explain in a minute.

vim /etc/default/shellinabox

And edit this line:

SHELLINABOX_ARGS="--no-beep -s /terminal:LOGIN --disable-ssl --localhost-only"

Finally, restart the deamon:

/etc/init.d/shellinabox restart

And check if all went well:

/etc/init.d/shellinabox status

You should see:

Shell In A Box Daemon is running

Another way to verify is to check the open ports:

netstat -ntl

You should see:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State 
tcp 0 0 127.0.0.1:4200 0.0.0.0:* LISTEN


Setting up Lighttpd as a proxy

Shell In A Box runs on port 4200 by default. Although this can be changed to a more common 80 or even 443, this is not what I want. I decided to integrate it with another webserver, to be able to combine other services and use just one url (and one SSL certificate). Since the Raspberry Pi isn’t that powerful, I choose Lighttpd.

apt-get instal lighttpd
cd /etc/lighttpd/conf-enabled
ln -s ../conf-available/10-proxy.conf

This installs Lighttpd and enables Proxy support. Now add the Proxy config:

vim /etc/lighttpd/lighttpd.conf

And add:

proxy.server = (
 "/terminal" =>
  ( (
    "host" => "127.0.0.1",
    "port" => 4200
  ) )
)

Save and restart Lighttpd:

/etc/init.d/lighttpd restart

Connect to http://pi.example.org/terminal and your Shell In A Box terminal should appear.

Although this is cool already, we’re not quite there. No one will SSH on an unencrypted web page, right? So, we’ll configure an SSL certificate to enable encryption. For double safety, we’ll also set a username/password on the web page. One then needs to know this password to access the login promt, and needs a valid local username/password to really use the terminal.

Adding encryption with SSL
By using a HTTPS-url, our traffic is encrypted. Let’s generate a private key (and remove the passphrase):

openssl genrsa -des3 -out pi.example.org.key 2048
cp -pr pi.example.org.key pi.example.org.key.passwd
openssl rsa -in pi.example.org.key.passwd -out pi.example.org.key

If you do not remove the passphrase, you will need to type it every time you start the web server. To request a SSL-certificate, you need to supply a CSR (Certificate Signing Request) and send that to a SSL provider such as Thawte or Verisign.

openssl req -new -key pi.example.org.key -out pi.example.org.csr

To be able to continue now, let’s self-sign the certificate:

openssl x509 -in pi.example.org.csr -out pi.example.org.pem -req -signkey pi.example.org.key -days 365
cat pi.example.org.key >> pi.example.org.pem

A self-signed certificate will display a warning in our browser, but that’s ok for now. Once the real certificate comes back from our SSL provider, it’s easy to replace it. The warning will then disappear.

Time to tell Lighttpd about our certificate:

vim /etc/lighttpd/lighttpd.conf

Add these lines:

$SERVER["socket"] == "10.0.0.10:443" {
  ssl.engine = "enable"
  ssl.pemfile = "/etc/lighttpd/ssl/pi.example.org/pi.example.org.pem"
  server.name = "pi.example.org"
  server.document-root = "/home/lighttpd/pi.example.org/https"
  server.errorlog = "/var/log/lighttpd/pi.example.org_serror.log"
  accesslog.filename = "/var/log/lighttpd/pi.example.org_saccess.log"
}

And restart Lighttpd:

/etc/init.d/lighttpd restart

Now Shell In A Box should be available on: https://pi.example.org/terminal

Enhancing security by adding HTTP-auth
Since the /terminal page now makes an actual terminal available to web users, I added an extra password for security. You can use the ‘HTTP Auth’ method for this. It will pop up a message box that requires an valid username/password before the /terminal page is shown.

First enable the module:

cd /etc/lighttpd/conf-enabled
ln -s ../conf-available/05-auth.conf

Then extend the config of the virtual host config you created above. The final result should be:

$SERVER["socket"] == "10.0.0.10:443" {
  ssl.engine = "enable"
  ssl.pemfile = "/etc/lighttpd/ssl/pi.example.org/pi.example.org.pem"

  server.name = "pi.example.org"
  server.document-root = "/home/lighttpd/pi.example.org/https"
  server.errorlog = "/var/log/lighttpd/pi.example.org_serror.log"
  accesslog.filename = "/var/log/lighttpd/pi.example.org_saccess.log"

  auth.debug = 2
  auth.backend = "htpasswd"
  auth.backend.htpasswd.userfile = "/etc/lighttpd/shellinabox-htpasswd"

  auth.require = ( "/terminal/" =>
    (
      "method" => "basic",
      "realm" => "Password protected area",
      "require" => "user=remibergsma"
    )
  )
}

Reload Lighttpd to make the changes active:

/etc/init.d/lighttpd reload

To set a password:

apt-get install apache2-utils
htpasswd -c -m /etc/lighttpd/shellinabox-htpasswd remibergsma

You can enter multiple users, just remember to remove the ‘-c’ flag when adding more users, as this overwrites the current file.

When you visit https://pi.example.org/terminal you will need to enter a valid username and password, before the page loads.

The final result: SSH in a browser window!
You should now be able to use a terminal via your own protected webpage. It’s mostly like a real terminal/SSH session but from a browser. Wow 🙂

Shell In A Box in action

Shell In A Box in action

 

I always use GNU Screen, so I know for sure my commands keep running whatever happens.

Using GNU Screen in a browser

Using GNU Screen in a browser