Archives For terminal

When a SSH-key changed, this warning is displayed:

ssh -l username 172.16.12.34

Warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
aa:bb:cc:a2:b6:87:bd:43:f9:ff:<wbr />02:8e:a6:b8:29:42.
Please contact your system administrator.
Add correct host key in /home/remi/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/remi/.ssh/known_hosts:6

In case you know the host identification has changed, you can safely discard this warning.

You could run ‘vim ~/.ssh/known_hosts‘ enter ‘6G‘ to go to the 6th line, ‘dd‘ to delete that line and finally ‘:wq‘ to save the file. But, wouldn’t a one-liner be handy?

Try:

ssh-keygen -R 172.24.111.132

Output:

/home/remi/.ssh/known_hosts updated.
Original contents retained as /home/remi/.ssh/known_hosts.

I’ve tested this on Ubuntu, Debian, SUSE and Red Hat Enterprise Linux.

In case you get this error:

fopen: No such file or directory

There isn’t a ‘known_hosts’ file in ‘~/.ssh/’. You can use the -f flag to specify the right file.

When you boot your Raspberry Pi with Raspbian, it will get its ip address from a DHCP server. Most tutorials I’ve seen, ask you to login to a ‘router’ (that serves as DHCP server as well) to find the assigned ip address. When you haven’t access to the DHCP server, how do you find the Pi’s ip address?
You should know the MAC address of a Raspberry Pi always has a ‘b8:27:eb’ prefix. Using a Linux utility called ‘arp-scan‘ you can discover MAC addresses and their corresponding ip addresses. Arp-scan sends ARP packets to hosts on the local network and displays any responses that are received.
First, install arp-scan:
apt-get install arp-scan
Then locate all Raspberry Pi’s on the local network using this command:
arp-scan --interface=eth0 --localnet | grep b8:27:eb
You will need to be root, or arp-scan must be SUID root, in order to run arp-scan, because the functions that it uses to read and write  packets require root privilege.
Result:
172.16.12.34 b8:27:eb:00:11:ab (Unknown)
Using the ip address, it is now possible to SSH into your Raspberry Pi:
ssh -l pi 172.16.12.34

Recently I was looking for a way to SSH from a network that blocked my outgoing SSH connection. I’d be nice to have a way around firewalls and be able to access your private Linux terminal. To be able to debug a problem from an remote location, for example.

A collegue suggested a tool called ‘Shell In A Box‘. Shell In A Box implements a web server that can export arbitrary command line tools to a web based terminal emulator using just JavaScript and CSS without any additional browser plugins. This means: connecting your browser via HTTPS to your own hosted Shell In A Box web site, and access a Linux terminal from there.

How cool is that? In this blog I’ll show you how to set it up in a secure way.

Building and installing Shell In A Box
I want to setup Shell In A Box on my Raspberry Pi. It’s a great device running Linux that has a very small energy consumption footprint. Ideal for an always-on device I’d say!

Since there is no package available, we’ve to compile our own. It’s best to get the sources from Github (original here), since the Github repository contains some patches and fixes for issues on Firefox.

These commands install the required dependencies, clone the Git repository and start building:

apt-get install git dpkg-dev debhelper autotools-dev libssl-dev libpam0g-dev zlib1g-dev libssl1.0.0 libpam0g openssl
git clone https://github.com/pythonanywhere/shellinabox_fork
cd shellinabox_fork
dpkg-buildpackage

During my first attempt, I ran into this problem:

dpkg-source -b shellinabox-2.14
dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../shellinabox_2.14.orig.tar.{bz2,gz,lzma,xz}
dpkg-buildpackage: error: dpkg-source -b shellinabox-2.14 gave error exit status 255

When grepping for ‘quilt’ I found a file called ‘/debian/source/format’. From what I can tell this does not do anything important, so I ended up deleting the file. Guess what, it now works.

rm ./debian/source/format

Build the package again, this should now succeed.

dpkg-buildpackage

This process will take some time (especially on the Raspberry Pi). Afterwards you’ll find the .deb file ready to be installed.

dpkg -i ../shellinabox_2.14-1_armhf.deb

I changed the configuration, to disallow the build-in SSL and to bind to localhost only. I did this because another web server will serve our terminal. I will explain in a minute.

vim /etc/default/shellinabox

And edit this line:

SHELLINABOX_ARGS="--no-beep -s /terminal:LOGIN --disable-ssl --localhost-only"

Finally, restart the deamon:

/etc/init.d/shellinabox restart

And check if all went well:

/etc/init.d/shellinabox status

You should see:

Shell In A Box Daemon is running

Another way to verify is to check the open ports:

netstat -ntl

You should see:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State 
tcp 0 0 127.0.0.1:4200 0.0.0.0:* LISTEN


Setting up Lighttpd as a proxy

Shell In A Box runs on port 4200 by default. Although this can be changed to a more common 80 or even 443, this is not what I want. I decided to integrate it with another webserver, to be able to combine other services and use just one url (and one SSL certificate). Since the Raspberry Pi isn’t that powerful, I choose Lighttpd.

apt-get instal lighttpd
cd /etc/lighttpd/conf-enabled
ln -s ../conf-available/10-proxy.conf

This installs Lighttpd and enables Proxy support. Now add the Proxy config:

vim /etc/lighttpd/lighttpd.conf

And add:

proxy.server = (
 "/terminal" =>
  ( (
    "host" => "127.0.0.1",
    "port" => 4200
  ) )
)

Save and restart Lighttpd:

/etc/init.d/lighttpd restart

Connect to http://pi.example.org/terminal and your Shell In A Box terminal should appear.

Although this is cool already, we’re not quite there. No one will SSH on an unencrypted web page, right? So, we’ll configure an SSL certificate to enable encryption. For double safety, we’ll also set a username/password on the web page. One then needs to know this password to access the login promt, and needs a valid local username/password to really use the terminal.

Adding encryption with SSL
By using a HTTPS-url, our traffic is encrypted. Let’s generate a private key (and remove the passphrase):

openssl genrsa -des3 -out pi.example.org.key 2048
cp -pr pi.example.org.key pi.example.org.key.passwd
openssl rsa -in pi.example.org.key.passwd -out pi.example.org.key

If you do not remove the passphrase, you will need to type it every time you start the web server. To request a SSL-certificate, you need to supply a CSR (Certificate Signing Request) and send that to a SSL provider such as Thawte or Verisign.

openssl req -new -key pi.example.org.key -out pi.example.org.csr

To be able to continue now, let’s self-sign the certificate:

openssl x509 -in pi.example.org.csr -out pi.example.org.pem -req -signkey pi.example.org.key -days 365
cat pi.example.org.key >> pi.example.org.pem

A self-signed certificate will display a warning in our browser, but that’s ok for now. Once the real certificate comes back from our SSL provider, it’s easy to replace it. The warning will then disappear.

Time to tell Lighttpd about our certificate:

vim /etc/lighttpd/lighttpd.conf

Add these lines:

$SERVER["socket"] == "10.0.0.10:443" {
  ssl.engine = "enable"
  ssl.pemfile = "/etc/lighttpd/ssl/pi.example.org/pi.example.org.pem"
  server.name = "pi.example.org"
  server.document-root = "/home/lighttpd/pi.example.org/https"
  server.errorlog = "/var/log/lighttpd/pi.example.org_serror.log"
  accesslog.filename = "/var/log/lighttpd/pi.example.org_saccess.log"
}

And restart Lighttpd:

/etc/init.d/lighttpd restart

Now Shell In A Box should be available on: https://pi.example.org/terminal

Enhancing security by adding HTTP-auth
Since the /terminal page now makes an actual terminal available to web users, I added an extra password for security. You can use the ‘HTTP Auth’ method for this. It will pop up a message box that requires an valid username/password before the /terminal page is shown.

First enable the module:

cd /etc/lighttpd/conf-enabled
ln -s ../conf-available/05-auth.conf

Then extend the config of the virtual host config you created above. The final result should be:

$SERVER["socket"] == "10.0.0.10:443" {
  ssl.engine = "enable"
  ssl.pemfile = "/etc/lighttpd/ssl/pi.example.org/pi.example.org.pem"

  server.name = "pi.example.org"
  server.document-root = "/home/lighttpd/pi.example.org/https"
  server.errorlog = "/var/log/lighttpd/pi.example.org_serror.log"
  accesslog.filename = "/var/log/lighttpd/pi.example.org_saccess.log"

  auth.debug = 2
  auth.backend = "htpasswd"
  auth.backend.htpasswd.userfile = "/etc/lighttpd/shellinabox-htpasswd"

  auth.require = ( "/terminal/" =>
    (
      "method" => "basic",
      "realm" => "Password protected area",
      "require" => "user=remibergsma"
    )
  )
}

Reload Lighttpd to make the changes active:

/etc/init.d/lighttpd reload

To set a password:

apt-get install apache2-utils
htpasswd -c -m /etc/lighttpd/shellinabox-htpasswd remibergsma

You can enter multiple users, just remember to remove the ‘-c’ flag when adding more users, as this overwrites the current file.

When you visit https://pi.example.org/terminal you will need to enter a valid username and password, before the page loads.

The final result: SSH in a browser window!
You should now be able to use a terminal via your own protected webpage. It’s mostly like a real terminal/SSH session but from a browser. Wow 🙂

Shell In A Box in action

Shell In A Box in action

 

I always use GNU Screen, so I know for sure my commands keep running whatever happens.

Using GNU Screen in a browser

Using GNU Screen in a browser

 

I’m using SSH’s key-based authentication to login to my servers using a SSH key pair. Of course my private key is protected with a password of its own. This has the consequence that even with key-based authentication, you still need to type a password: your private key passphrase. And you need to type it on every single connect. Although this is far more secure that having no passphrase on your private key, typing it so many times is a bit over done.

In this blog I’ll show you how to setup a system where you enter the passphrase only once per day when you start a new terminal session. For me this works great, as I now only type my passphrase in the morning when I login to my workstation and am able to connect password-less the rest of the day. Whenever I leave my desk, my screen is locked so no one can access my key.

Setting up key-based authentication

The first step is to generate a key pair to use.

ssh-keygen -t rsa -b 2048

The output is something like below. Enter your passphrase twice, when prompted.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/remi/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/remi/.ssh/id_rsa.
Your public key has been saved in /home/remi/.ssh/id_rsa.pub.
The key fingerprint is:
41:22:af:8b:4b:aa:94:34:a4:fa:5a:eb:91:4c:5b:5e remi@wks

The key's randomart image is:

+--[ RSA 2048]----+
|    . . .        |
|     o o         |
| .    . .        |
|o    .   .       |
|.o. o E S        |
|ooo* o           |
|.oO o            |
|.= +             |
|=o=              |
+-----------------+

To setup key-based authentication, copy your public key to the ‘.ssh/authorized_keys’ file on the server you want to connect to. You can do this by using ‘ssh-copy-id‘. You need to have access to the user account you want to connect to because you need to type its password to authenticate the copying.

ssh-copy-id -i ~/.ssh/id_rsa.pub user@server

You can now login to this server by typing only the passphrase of your private key, instead of typing the user account’s password.

Using ssh-agent to manage private keys
When you want to take this to the next level, you can use ‘ssh-agent‘ to manage your SSH keys. This program is started at login and remembers the keys you add each session. Through use of environment variables the agent can be located and automatically used for authentication when logging in to other machines using SSH.

The idea is that the agent is run in the user’s local PC, laptop, or terminal.  Authentication data need not be stored on any other machine, and authentication passphrases never go over the network. However, the connection to the agent is forwarded over SSH remote logins, and the user can thus use the privileges given by the identities anywhere in the network in a secure way.

To add a key, run ‘ssh-add‘ and type the passphrase of your private key. This is needed only once per session. After you’ve added your key to ‘ssh-agent‘, you should be able to SSH to the local machine without entering a password. This is more secure than a key pair without a passphrase because whenever your key gets lost, one still needs to type your passphrase to get access.

Use case
At work, I SSH to a central server from which I can access all the other servers. My public key is distributed to all servers, my private key is on my workstation. Now, for secure password-less logins to work, I need to be able to access my local ssh-agent from this central server. When you SSH to your server as I describe above, you don’t have to type the passphrase. When you have a look at the environment variables, you’ll notice SSH_AUTH_SOCK is set. This variable holds the path to the socket connecting to your ‘ssh-agent‘ running on your local machine. Authentication is now done this way!

Getting this to work with GNU screen
There’s a little problem when using this technique with GNU screen. When you reconnect to a running screen session from another location, the environment variable inside screen is not updated and so you need to type the password for your private key on each connect again. I found this post that describes a solution, which I further enhanced to make it work correctly for me:

export SSH_AUTH_SOCK=$(find /tmp/ssh-* -user `whoami` -name agent\* -printf '%T@ %p\n' 2>/dev/null | sort -k 1nr | sed 's/^[^ ]* //' | head -n 1)

This one-liner looks for the most recent ssh-agent socket, owned by the current user and sets the SSH_AUTH_SOCK environment variable to this socket. You can now use this socket again to authenticate to your locally running ‘ssh-agent‘.

To make this a little easier, I added this scrip ‘update-ssh-agent-socket.sh‘ on my home directory on the central server (also available on github):

#!/bin/bash

echo "Updating ssh-agent socket environment.."
echo "Current value:  $SSH_AUTH_SOCK"

export SSH_AUTH_SOCK=$(find /tmp/ssh-* -user `whoami` -name agent\* -printf '%T@ %p\n' 2>/dev/null | sort -k 1nr | sed 's/^[^ ]* //' | head -n 1)

if [ $? -gt 0 ]; then
 echo "ERROR!"
 exit $?
fi

echo "New value:  $SSH_AUTH_SOCK"
echo "All done!"

Run it like this to use the most recent ssh-agent socket (note the dot at the start):

. ~/update-ssh-agent-socket.sh

Usually I type ‘<CTR>+r update <enter>’ to quickly run this command from history. This is the quickest way I could think of to get everything working and make it easy to quickly login to our servers.

I’ve upgraded my 3 year old MBP with a fast SSD. I decided to install from scratch instead of migrating or cloning the old disk. The good thing is that everything is now fast and clean. The downside is that all customisations I did over the past years are also gone.

One thing I customised was shortcuts. I forgot how easy this can be done in OSX, so I’ll describe it below so others can benefit as well.

For example, I want the shortcuts in Terminal for Next and Previous tab to be CMD+right arrow, and CMD+left arrow. In Terminal you can see the default shortcuts for this in the menu:

It looks ok, but you cannot directly press ‘}’ on most keyboards; It’s SHIFT+], so this makes the default shortcut CMD+SHIFT+]. I prefer a shortcut with just two keys. To change these you’d go to:

System Preferences | Keyboard | Keyboard Shortcuts

From the left, choose ‘Application Shortcuts’ and then click the + icon to add a new shortcut. First, select the application you want to change the shortcut for, ‘Terminal.app’ in this example. Then, specify the exact menu title for the command you’d like to change the shortcut for. As you can see above, you’d enter ‘Select Next Tab’ to change that command. Repeat for ‘Select Previous Tab’. Just press the shortcut and it will appear in the box. Afterwards it should look like this in the keyboard preferences pane:

Now, go back to the Terminal application and have a look at the same menu as before. It should now list the shortcut you just added.

Try changing tabs by pressing CMD+left/right arrow.. it works!

One thing to note, it that the shortcuts I used in this example, by default belong to changing windows instead of tabs. When you want to change windows in Terminal using a shortcut, you’ll need to add new shortcuts for these as well. Alternatively, you use the CMS+` shortcut which still cycles through your windows, it just cannot go back. For me this works ok as this works in any application.

Changing shortcuts in Mac OSX is easy and allows you to make them work exactly as you prefer. I’ve done this in Mountain Lion, but I believe this can be done in Lion as well.

I’ve had some trouble when using the Mac OSX Terminal app for some time now. Until today, it just gave me some annoying warnings from time to time. Like when installing an application with apt-get in Debian:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = “UTF-8”,
LANG = “en_US.UTF-8”
are supported and installed on your system.
perl: warning: Falling back to the standard locale (“C”).
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

It did work, so nothing too serious. I’ve also found applications, like iotop for example, that refuses to start when LC_ALL was unset. But a quick

EXPORT LC_ALL=$LANG

made the application start, so I didn’t take the time to investigate it further. Today I run into a more serious issue that cost me quite some time to figure out.

I had stopped the pure-ftpd deamon to do some maintenance and then started it again. It did start without error, but connecting failed:

server:~# ftp ftp.server.nl
Connected to ftp.server.nl.
perl: warning: Setting locale failed.
ftp>

Nothing had changed in the ftp configuration. After some debugging and trial & error, I found out that when I started the deamon from within a shell on Ubuntu it worked, but when I started it within a shell on my MacBook, it didn’t.

When looking at the locales I found:

server:~# locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LC_CTYPE=UTF-8
LC_NUMERIC=”en_US.UTF-8″
LC_TIME=”en_US.UTF-8″
LC_COLLATE=”en_US.UTF-8″
LC_MONETARY=”en_US.UTF-8″
LC_MESSAGES=”en_US.UTF-8″
LC_PAPER=”en_US.UTF-8″
LC_NAME=”en_US.UTF-8″
LC_ADDRESS=”en_US.UTF-8″
LC_TELEPHONE=”en_US.UTF-8″
LC_MEASUREMENT=”en_US.UTF-8″
LC_IDENTIFICATION=”en_US.UTF-8″
LC_ALL=

Notice the two errors at the top. I talked to a colleague about this and he suggested looking at the Terminal app settings. There I found a setting called “Set locale environment variables on startup” which was activated. The setting is located in Preferences | Settings | Advanced. I’ve unchecked the button now as you can see in this screenshot:

When closing the Terminal app, and reopening it again, I tried again:

server:~# locale
LANG=en_US.UTF-8
LC_CTYPE=”en_US.UTF-8″
LC_NUMERIC=”en_US.UTF-8″
LC_TIME=”en_US.UTF-8″
LC_COLLATE=”en_US.UTF-8″
LC_MONETARY=”en_US.UTF-8″
LC_MESSAGES=”en_US.UTF-8″
LC_PAPER=”en_US.UTF-8″
LC_NAME=”en_US.UTF-8″
LC_ADDRESS=”en_US.UTF-8″
LC_TELEPHONE=”en_US.UTF-8″
LC_MEASUREMENT=”en_US.UTF-8″
LC_IDENTIFICATION=”en_US.UTF-8″
LC_ALL=

No more errors! I tried restarting the pure-ftpd deamon from my Terminal app and it now works as expected. Even the warnings and errors when installing applications in Debian (apt-get) are gone. In fact, it seems this is the way it is supposed to work.

Glad I’ve fixed this 🙂

Update: As Reza mentions in the comments, it’s also possible to fix this problem on the server side. This is the best way to go if you want to fix this for your users. Thanks Reza!

Whether you’re using Mac OSX, Windows or Linux, we’re all using a so-called “window manager”. Most are graphical user interfaces; and that’s a good thing 🙂 But as a sysadmin I need to manage many servers. Servers without a graphical user interface. So, how to handle that?

One could just ssh into a server when you need to do work on a given server. Depending on the terminal program you use, you might be able to have multiple sessions at the same time, preferably in tabs. It looks like this when 3 tabs are open:

Although this works well, it has one drawback for me: it only works on one computer. And when you turn off your computer everything is gone and disconnected. Since I’m working on multiple computers (desktop/laptop), multiple OS’es (OSX, Ubuntu) and multiple places (Work, Home), this no longer worked for me and I started looking for a better solution.

GNU Screen to the rescue! GNU Screen is a full-screen window manager, but terminal based. That is, it works in interactive shells such as a ssh session and is able to keep running while disconnected.

Starting a screen is easy:

screen -S screenname

You can attach and detach a session when needed. To detach, press Ctrl+a+d. To reattach enter:

screen -r screenname

This means that no matter on what computer I login, on any place, I always am able to attach a running screen. It looks like this:

Note the bar at the bottom where the tabs are. You can even give them a name!

GNU Screen can be a bit obscure to configure. After googling a lot and some help from co-workers, I have now configured GNU Screen as you can see on the above image. Configuration is read from the .screenrc file in your homedir. In the image above you can see my .screenrc file.

Commands in GNU Screen are all prefixed by a control command, the default is Ctrl+a. This means that all commands you type will be entered in the terminal you’re connected to, except commands followed directly after you press Ctrl+a. In the manual page you’ll find C-a, which is short for Ctrl+a.

When you want to create a new tab, enter Ctrl+a+c. To change from one tab to the other, you press Ctrl+a+2 to go to tab #2. Ctrl+a+spacebar brings you to the next tab. To name that tab, enter Ctrl+a+A, etc. It takes a bit of time to get used to it, but for me it works very well.

GNU Screen is a window manager, so apart from multiple tabs, you can even split the window to host multiple screens next to each other. To split vertically you enter Ctrl+a+|. Ctrl+a+tab brings you to the newly created space. You then create a new tab there, like Ctrl+a+c. It looks like this:

Like I said, Ctrl+a+tab switches between the left and right screen. In each you can call any tab that is below with Ctrl+a+tabnumner, or toggle between them using the spacebar.

Need to go away and want to protect your screens? Enter Ctrl+a+x and you’re screens will be locked. You need to enter the password of the connected user to unlock.

This is just a quick introduction of what is possible with screen. Have a look at the man page:

man screen

or use Google to get some more configuration examples. Have fun!