Archives For ssh

As discussed in a previous post, multi-factor authentication really makes things more secure. Let’s see how we can secure services like SSH and the Gnome desktop with multi-factor authentication.

The Google Authenticator project provides a PAM module than can be integrated with your Linux server or desktop. This PAM module is designed for for home and other small environments. There is no central management of keys, and the configuration is saved in each users home folder. I’ve successfully deployed this on my home server (a Raspberry Pi) and on my work laptop.

When using a Debian-like OS, you can install it with a one-liner

apt-get install libpam-google-authenticator

But note the packaged version is old and does not support all documented options. Below I talk about the ‘nullok’ option, but that is not supported in the packaged version. You then see this error:

sshd(pam_google_authenticator)[884]: Unrecognized option "nullok"
sshd(pam_google_authenticator)[887]: Failed to read "/home/pi/.google_authenticator"

That’s why I suggest building from source, as this can be done quickly:

apt-get remove libpam-google-authenticator
apt-get install libpam0g-dev libqrencode3
cd ~
wget https://google-authenticator.googlecode.com/files/libpam-google-authenticator-1.0-source.tar.bz2
tar jvxf libpam-google-authenticator-1.0-source.tar.bz2
cd libpam-google-authenticator-1.0/
make
make install

Open a shell for a user you want to enable two-factor authentication for and run ‘google-authenticator’ to configure.

google-authenticator
Configure Google Authenticator

Configure Google Authenticator

Nice!

Configure your Mobile device
Install the ‘Google Authenticator’ app and just scan the QR-code. It will automatically configure itself and start displaying verification codes.

Verification codes displayed by app

Verification codes displayed by app

You should notice it’ll display new codes each 30 seconds.

Configure SSH
Two files need to be edited in order to enable two-factor authentication in SSH.

vim /etc/pam.d/sshd

Add this line:

auth required pam_google_authenticator.so nullok

Where to put this in the file? That depends. When you put it at the top, SSH will first ask a verification code, then a password. To me this sounds unlogical, so I placed it just below this line:

@include common-auth

The ‘nullok’ option, by the way, tells PAM whenever no config for 2-factor authentication is found, it should just ignore it. If you want SSH logins to fail, when no two-factor authentication is configured, you can delete the option. Be warned to at least config it for one user, or you will be locked out of your server.

Now tel SSH to ask for the verification code:

vim /etc/ssh/sshd_config

Edit the setting, it’s probably set to ‘no’:

ChallengeResponseAuthentication yes

Now all you need to do is restart SSH. Keep a spare SSH session logged-in, just in case.

/etc/init.d/ssh restart
SSH 2-factor authentication in action

SSH two-factor authentication in action

SSH will now ask for a verification code when you do an interactive login. When a certificate is used, no verification code is asked.

Configuring Ubuntu Desktop
The Ubuntu desktop can also ask for a verification code. Just edit one file:

vim /etc/pam.d/lightdm

And add this line, just like SSH:

auth required pam_google_authenticator.so nullok

The login screen should now ask for a verification code. It looks like:

Two-factor authentication in Ubuntu

Two-factor authentication in Ubuntu

You can setup any PAM service like this, the basic principles are all the same.

Skip two-factor authentication if logging in from the local network
At first this is all very cool, but soon it becomes a bit annoying, too. When I SSH from a local network, I just don’t want to enter the verification code because I trust my local network. When I SSH from remote, a verification code is required. One way to arrange that, is always login with certificates. But there is another way to configure it: using the pam_access module. Try this config:

auth [success=1 default=ignore] pam_access.so accessfile=/etc/security/access-local.conf
auth required pam_google_authenticator.so nullok

The config file, /etc/security/access-local.conf looks like:

# Two-factor can be skipped on local network
 + : ALL : 10.0.0.0/24
 + : ALL : LOCAL
 - : ALL : ALL

Local login attempts from 10.0.0.0/24 will not require two-factor authentication, while all others do.

Two-factor only for users of a certain group
You can even enable two-factor authentication only for users of a certain group. Say, you’d want only users in the ‘sudo’ group to do use two-factor authentication. You could use:

auth [success=1 default=ignore] pam_succeed_if.so user notingroup sudo
auth required pam_google_authenticator.so nullok

By the way, “[success=1 default=ignore]” in PAM means: on success, skip the next 1 auth provider (two-factor authentication), on failure, just pretend it didn’t happen.

Moving the config outside the user’s home
Some people prefer not to store the authentication data in the user’s home directory. You can do that using the ‘secret’ parameter:

mkdir -p /var/lib/google-authenticator/
mv ~remibergsma/.google_authenticator /var/lib/google-authenticator/remibergsma
chown root.root /var/lib/google-authenticator/*

The PAM recipe will then become:

auth required pam_google_authenticator.so nullok user=root secret=/var/lib/google-authenticator/${USER}

As you’ve seen, PAM is very flexible. You can write almost anything you want.

Try the Google Authenticator two-factor authentication. It’s free and adds another security layer to your home server.

When it not possible to reach a server you want to SSH to directly, you can make use of SSH’s built in capability to chain multiple commands. Suppose you have a network setup like in the image below.

Network overview

Network overview

Firewalls or ACL’s prevent direct access to the ‘web server’ in network #2. In between is a ‘jump host’ in network #1. A ‘jump host’ is a host you can SSH to, and from there reach the next hop. How to SSH to the web server?

You could do this manually:

ssh -l user jump-host

and then from that server:

ssh -l user webserver.dmz

But using the -t switch, you can chain them together like this:

ssh -A -t -l user jump-host \
ssh -A -t -l user webserver.dmz

The -A switch enables forwarding of the ssh-agent. When using key based authentication, you’ll be able to login with typing the certificate’s password only once.

Using this technique, you can also build a SSH tunnel through the jump host:

ssh -A -t -l user jump-host \
-L 8080:localhost:8080 \
ssh -A -t -l user webserver.dmz \
-L 8080:localhost:8080

When you type: http://localhost:8080 in a browser, you are connected over a secure tunnel to the web server in Network #2. Thanks to the chaining of commands, this is now possible.

You can use many chained commands, so this is very flexible.

When a SSH-key changed, this warning is displayed:

ssh -l username 172.16.12.34

Warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
aa:bb:cc:a2:b6:87:bd:43:f9:ff:<wbr />02:8e:a6:b8:29:42.
Please contact your system administrator.
Add correct host key in /home/remi/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/remi/.ssh/known_hosts:6

In case you know the host identification has changed, you can safely discard this warning.

You could run ‘vim ~/.ssh/known_hosts‘ enter ‘6G‘ to go to the 6th line, ‘dd‘ to delete that line and finally ‘:wq‘ to save the file. But, wouldn’t a one-liner be handy?

Try:

ssh-keygen -R 172.24.111.132

Output:

/home/remi/.ssh/known_hosts updated.
Original contents retained as /home/remi/.ssh/known_hosts.

I’ve tested this on Ubuntu, Debian, SUSE and Red Hat Enterprise Linux.

In case you get this error:

fopen: No such file or directory

There isn’t a ‘known_hosts’ file in ‘~/.ssh/’. You can use the -f flag to specify the right file.

Recently I was looking for a way to SSH from a network that blocked my outgoing SSH connection. I’d be nice to have a way around firewalls and be able to access your private Linux terminal. To be able to debug a problem from an remote location, for example.

A collegue suggested a tool called ‘Shell In A Box‘. Shell In A Box implements a web server that can export arbitrary command line tools to a web based terminal emulator using just JavaScript and CSS without any additional browser plugins. This means: connecting your browser via HTTPS to your own hosted Shell In A Box web site, and access a Linux terminal from there.

How cool is that? In this blog I’ll show you how to set it up in a secure way.

Building and installing Shell In A Box
I want to setup Shell In A Box on my Raspberry Pi. It’s a great device running Linux that has a very small energy consumption footprint. Ideal for an always-on device I’d say!

Since there is no package available, we’ve to compile our own. It’s best to get the sources from Github (original here), since the Github repository contains some patches and fixes for issues on Firefox.

These commands install the required dependencies, clone the Git repository and start building:

apt-get install git dpkg-dev debhelper autotools-dev libssl-dev libpam0g-dev zlib1g-dev libssl1.0.0 libpam0g openssl
git clone https://github.com/pythonanywhere/shellinabox_fork
cd shellinabox_fork
dpkg-buildpackage

During my first attempt, I ran into this problem:

dpkg-source -b shellinabox-2.14
dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../shellinabox_2.14.orig.tar.{bz2,gz,lzma,xz}
dpkg-buildpackage: error: dpkg-source -b shellinabox-2.14 gave error exit status 255

When grepping for ‘quilt’ I found a file called ‘/debian/source/format’. From what I can tell this does not do anything important, so I ended up deleting the file. Guess what, it now works.

rm ./debian/source/format

Build the package again, this should now succeed.

dpkg-buildpackage

This process will take some time (especially on the Raspberry Pi). Afterwards you’ll find the .deb file ready to be installed.

dpkg -i ../shellinabox_2.14-1_armhf.deb

I changed the configuration, to disallow the build-in SSL and to bind to localhost only. I did this because another web server will serve our terminal. I will explain in a minute.

vim /etc/default/shellinabox

And edit this line:

SHELLINABOX_ARGS="--no-beep -s /terminal:LOGIN --disable-ssl --localhost-only"

Finally, restart the deamon:

/etc/init.d/shellinabox restart

And check if all went well:

/etc/init.d/shellinabox status

You should see:

Shell In A Box Daemon is running

Another way to verify is to check the open ports:

netstat -ntl

You should see:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State 
tcp 0 0 127.0.0.1:4200 0.0.0.0:* LISTEN


Setting up Lighttpd as a proxy

Shell In A Box runs on port 4200 by default. Although this can be changed to a more common 80 or even 443, this is not what I want. I decided to integrate it with another webserver, to be able to combine other services and use just one url (and one SSL certificate). Since the Raspberry Pi isn’t that powerful, I choose Lighttpd.

apt-get instal lighttpd
cd /etc/lighttpd/conf-enabled
ln -s ../conf-available/10-proxy.conf

This installs Lighttpd and enables Proxy support. Now add the Proxy config:

vim /etc/lighttpd/lighttpd.conf

And add:

proxy.server = (
 "/terminal" =>
  ( (
    "host" => "127.0.0.1",
    "port" => 4200
  ) )
)

Save and restart Lighttpd:

/etc/init.d/lighttpd restart

Connect to http://pi.example.org/terminal and your Shell In A Box terminal should appear.

Although this is cool already, we’re not quite there. No one will SSH on an unencrypted web page, right? So, we’ll configure an SSL certificate to enable encryption. For double safety, we’ll also set a username/password on the web page. One then needs to know this password to access the login promt, and needs a valid local username/password to really use the terminal.

Adding encryption with SSL
By using a HTTPS-url, our traffic is encrypted. Let’s generate a private key (and remove the passphrase):

openssl genrsa -des3 -out pi.example.org.key 2048
cp -pr pi.example.org.key pi.example.org.key.passwd
openssl rsa -in pi.example.org.key.passwd -out pi.example.org.key

If you do not remove the passphrase, you will need to type it every time you start the web server. To request a SSL-certificate, you need to supply a CSR (Certificate Signing Request) and send that to a SSL provider such as Thawte or Verisign.

openssl req -new -key pi.example.org.key -out pi.example.org.csr

To be able to continue now, let’s self-sign the certificate:

openssl x509 -in pi.example.org.csr -out pi.example.org.pem -req -signkey pi.example.org.key -days 365
cat pi.example.org.key >> pi.example.org.pem

A self-signed certificate will display a warning in our browser, but that’s ok for now. Once the real certificate comes back from our SSL provider, it’s easy to replace it. The warning will then disappear.

Time to tell Lighttpd about our certificate:

vim /etc/lighttpd/lighttpd.conf

Add these lines:

$SERVER["socket"] == "10.0.0.10:443" {
  ssl.engine = "enable"
  ssl.pemfile = "/etc/lighttpd/ssl/pi.example.org/pi.example.org.pem"
  server.name = "pi.example.org"
  server.document-root = "/home/lighttpd/pi.example.org/https"
  server.errorlog = "/var/log/lighttpd/pi.example.org_serror.log"
  accesslog.filename = "/var/log/lighttpd/pi.example.org_saccess.log"
}

And restart Lighttpd:

/etc/init.d/lighttpd restart

Now Shell In A Box should be available on: https://pi.example.org/terminal

Enhancing security by adding HTTP-auth
Since the /terminal page now makes an actual terminal available to web users, I added an extra password for security. You can use the ‘HTTP Auth’ method for this. It will pop up a message box that requires an valid username/password before the /terminal page is shown.

First enable the module:

cd /etc/lighttpd/conf-enabled
ln -s ../conf-available/05-auth.conf

Then extend the config of the virtual host config you created above. The final result should be:

$SERVER["socket"] == "10.0.0.10:443" {
  ssl.engine = "enable"
  ssl.pemfile = "/etc/lighttpd/ssl/pi.example.org/pi.example.org.pem"

  server.name = "pi.example.org"
  server.document-root = "/home/lighttpd/pi.example.org/https"
  server.errorlog = "/var/log/lighttpd/pi.example.org_serror.log"
  accesslog.filename = "/var/log/lighttpd/pi.example.org_saccess.log"

  auth.debug = 2
  auth.backend = "htpasswd"
  auth.backend.htpasswd.userfile = "/etc/lighttpd/shellinabox-htpasswd"

  auth.require = ( "/terminal/" =>
    (
      "method" => "basic",
      "realm" => "Password protected area",
      "require" => "user=remibergsma"
    )
  )
}

Reload Lighttpd to make the changes active:

/etc/init.d/lighttpd reload

To set a password:

apt-get install apache2-utils
htpasswd -c -m /etc/lighttpd/shellinabox-htpasswd remibergsma

You can enter multiple users, just remember to remove the ‘-c’ flag when adding more users, as this overwrites the current file.

When you visit https://pi.example.org/terminal you will need to enter a valid username and password, before the page loads.

The final result: SSH in a browser window!
You should now be able to use a terminal via your own protected webpage. It’s mostly like a real terminal/SSH session but from a browser. Wow 🙂

Shell In A Box in action

Shell In A Box in action

 

I always use GNU Screen, so I know for sure my commands keep running whatever happens.

Using GNU Screen in a browser

Using GNU Screen in a browser

 

I’m using SSH’s key-based authentication to login to my servers using a SSH key pair. Of course my private key is protected with a password of its own. This has the consequence that even with key-based authentication, you still need to type a password: your private key passphrase. And you need to type it on every single connect. Although this is far more secure that having no passphrase on your private key, typing it so many times is a bit over done.

In this blog I’ll show you how to setup a system where you enter the passphrase only once per day when you start a new terminal session. For me this works great, as I now only type my passphrase in the morning when I login to my workstation and am able to connect password-less the rest of the day. Whenever I leave my desk, my screen is locked so no one can access my key.

Setting up key-based authentication

The first step is to generate a key pair to use.

ssh-keygen -t rsa -b 2048

The output is something like below. Enter your passphrase twice, when prompted.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/remi/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/remi/.ssh/id_rsa.
Your public key has been saved in /home/remi/.ssh/id_rsa.pub.
The key fingerprint is:
41:22:af:8b:4b:aa:94:34:a4:fa:5a:eb:91:4c:5b:5e remi@wks

The key's randomart image is:

+--[ RSA 2048]----+
|    . . .        |
|     o o         |
| .    . .        |
|o    .   .       |
|.o. o E S        |
|ooo* o           |
|.oO o            |
|.= +             |
|=o=              |
+-----------------+

To setup key-based authentication, copy your public key to the ‘.ssh/authorized_keys’ file on the server you want to connect to. You can do this by using ‘ssh-copy-id‘. You need to have access to the user account you want to connect to because you need to type its password to authenticate the copying.

ssh-copy-id -i ~/.ssh/id_rsa.pub user@server

You can now login to this server by typing only the passphrase of your private key, instead of typing the user account’s password.

Using ssh-agent to manage private keys
When you want to take this to the next level, you can use ‘ssh-agent‘ to manage your SSH keys. This program is started at login and remembers the keys you add each session. Through use of environment variables the agent can be located and automatically used for authentication when logging in to other machines using SSH.

The idea is that the agent is run in the user’s local PC, laptop, or terminal.  Authentication data need not be stored on any other machine, and authentication passphrases never go over the network. However, the connection to the agent is forwarded over SSH remote logins, and the user can thus use the privileges given by the identities anywhere in the network in a secure way.

To add a key, run ‘ssh-add‘ and type the passphrase of your private key. This is needed only once per session. After you’ve added your key to ‘ssh-agent‘, you should be able to SSH to the local machine without entering a password. This is more secure than a key pair without a passphrase because whenever your key gets lost, one still needs to type your passphrase to get access.

Use case
At work, I SSH to a central server from which I can access all the other servers. My public key is distributed to all servers, my private key is on my workstation. Now, for secure password-less logins to work, I need to be able to access my local ssh-agent from this central server. When you SSH to your server as I describe above, you don’t have to type the passphrase. When you have a look at the environment variables, you’ll notice SSH_AUTH_SOCK is set. This variable holds the path to the socket connecting to your ‘ssh-agent‘ running on your local machine. Authentication is now done this way!

Getting this to work with GNU screen
There’s a little problem when using this technique with GNU screen. When you reconnect to a running screen session from another location, the environment variable inside screen is not updated and so you need to type the password for your private key on each connect again. I found this post that describes a solution, which I further enhanced to make it work correctly for me:

export SSH_AUTH_SOCK=$(find /tmp/ssh-* -user `whoami` -name agent\* -printf '%T@ %p\n' 2>/dev/null | sort -k 1nr | sed 's/^[^ ]* //' | head -n 1)

This one-liner looks for the most recent ssh-agent socket, owned by the current user and sets the SSH_AUTH_SOCK environment variable to this socket. You can now use this socket again to authenticate to your locally running ‘ssh-agent‘.

To make this a little easier, I added this scrip ‘update-ssh-agent-socket.sh‘ on my home directory on the central server (also available on github):

#!/bin/bash

echo "Updating ssh-agent socket environment.."
echo "Current value:  $SSH_AUTH_SOCK"

export SSH_AUTH_SOCK=$(find /tmp/ssh-* -user `whoami` -name agent\* -printf '%T@ %p\n' 2>/dev/null | sort -k 1nr | sed 's/^[^ ]* //' | head -n 1)

if [ $? -gt 0 ]; then
 echo "ERROR!"
 exit $?
fi

echo "New value:  $SSH_AUTH_SOCK"
echo "All done!"

Run it like this to use the most recent ssh-agent socket (note the dot at the start):

. ~/update-ssh-agent-socket.sh

Usually I type ‘<CTR>+r update <enter>’ to quickly run this command from history. This is the quickest way I could think of to get everything working and make it easy to quickly login to our servers.

Whether you’re using Mac OSX, Windows or Linux, we’re all using a so-called “window manager”. Most are graphical user interfaces; and that’s a good thing 🙂 But as a sysadmin I need to manage many servers. Servers without a graphical user interface. So, how to handle that?

One could just ssh into a server when you need to do work on a given server. Depending on the terminal program you use, you might be able to have multiple sessions at the same time, preferably in tabs. It looks like this when 3 tabs are open:

Although this works well, it has one drawback for me: it only works on one computer. And when you turn off your computer everything is gone and disconnected. Since I’m working on multiple computers (desktop/laptop), multiple OS’es (OSX, Ubuntu) and multiple places (Work, Home), this no longer worked for me and I started looking for a better solution.

GNU Screen to the rescue! GNU Screen is a full-screen window manager, but terminal based. That is, it works in interactive shells such as a ssh session and is able to keep running while disconnected.

Starting a screen is easy:

screen -S screenname

You can attach and detach a session when needed. To detach, press Ctrl+a+d. To reattach enter:

screen -r screenname

This means that no matter on what computer I login, on any place, I always am able to attach a running screen. It looks like this:

Note the bar at the bottom where the tabs are. You can even give them a name!

GNU Screen can be a bit obscure to configure. After googling a lot and some help from co-workers, I have now configured GNU Screen as you can see on the above image. Configuration is read from the .screenrc file in your homedir. In the image above you can see my .screenrc file.

Commands in GNU Screen are all prefixed by a control command, the default is Ctrl+a. This means that all commands you type will be entered in the terminal you’re connected to, except commands followed directly after you press Ctrl+a. In the manual page you’ll find C-a, which is short for Ctrl+a.

When you want to create a new tab, enter Ctrl+a+c. To change from one tab to the other, you press Ctrl+a+2 to go to tab #2. Ctrl+a+spacebar brings you to the next tab. To name that tab, enter Ctrl+a+A, etc. It takes a bit of time to get used to it, but for me it works very well.

GNU Screen is a window manager, so apart from multiple tabs, you can even split the window to host multiple screens next to each other. To split vertically you enter Ctrl+a+|. Ctrl+a+tab brings you to the newly created space. You then create a new tab there, like Ctrl+a+c. It looks like this:

Like I said, Ctrl+a+tab switches between the left and right screen. In each you can call any tab that is below with Ctrl+a+tabnumner, or toggle between them using the spacebar.

Need to go away and want to protect your screens? Enter Ctrl+a+x and you’re screens will be locked. You need to enter the password of the connected user to unlock.

This is just a quick introduction of what is possible with screen. Have a look at the man page:

man screen

or use Google to get some more configuration examples. Have fun!