Archives For shell

Recently a colleague came by and asked whether I could help him with finding out how much memory was already assigned to Oracle databases. It was fun to find out what was exactly he needed and to construct the one-liner bit by bit, as I detail below. When it seemed to work ok, I wrapped it in a small shell script to make it more flexible and reusable and deployed it to all Oracle database servers using CFEngine. You can find the final shell script on Github. I’d suggest you use the shell script instead of the one-liners below, as it is more accurate.

But let’s now play a bit with the shell. Since part of the memory used by Oracle is dynamically assigned, using tools like ‘free‘ will not really help to determine how much memory is allocated. My colleague told me we’d to look at the ‘PGA‘ (Program Global Area) and ‘SGA‘ (System Global Area) values in the Oracle settings of each database.

From the Oracle docs:

The SGA is a group of shared memory structures, known as SGA components, that contain data and control information for one Oracle Database instance. The SGA is shared by all server and background processes. Examples of data stored in the SGA include cached data blocks and shared SQL areas.

A PGA is a memory region that contains data and control information for a server process. It is nonshared memory created by Oracle Database when a server process is started. Access to the PGA is exclusive to the server process. There is one PGA for each server process. Background processes also allocate their own PGAs. The total memory used by all individual PGAs is known as the total instance PGA memory, and the collection of individual PGAs is referred to as the total instance PGA, or just instance PGA. You use database initialization parameters to set the size of the instance PGA, not individual PGAs.

These settings can be found in the ‘spfile*.ora‘ of a given database. This is how I found all of them:

find /oracle/product/*/db_1/dbs/spfile*.ora

Because these are binary data files, so you need ‘strings‘ to get the readable out put. The settings we were looking for had ‘target‘ in their name, so we could grep them out like this:

strings -a /oracle/product/11.x.y.z/db_1/dbs/spfiledbname.ora |\
grep -i target

We’re especially interested in settings that started with ‘*.pga‘ or ‘*.sga‘. Example:

*.pga_aggregate_target=955252736
 *.sga_target=1610612736

Of course, grep can do that too. On the lines we found, the value of the setting (next to the ‘=’) can be in kilobytes (as shown above), but also in megabytes (ending in ‘M’ or ‘m’) or gigabytes (ending in ‘G’ or ‘g’).

The question was: How to quickly sum these values on the command line to get an impression of the assigned memory. My colleagues usually did this by hand, but now there were a bit too many. You shouldn’t do this by hand anyway, if you ask me. So I gave it a try:

First we need to select the correct lines of all ‘spfiles‘ of all databases on a server:

find /oracle/product/*/db_1/dbs/spfile*.ora |\
xargs strings -a |\
grep target |\
grep -iE '\*.[ps]ga'

This resulted a list of all the settings we were looking for. The same as above, but for all databases, one per row. The next thing to do, to be able to sum them up, is to remove everything except the values we want to sum. The command ‘cut’ can do this, using a delimeter ‘=’ and then the second field. It looks like this:

find /oracle/product/*/db_1/dbs/spfile*.ora |\
xargs strings -a |\
grep target |\
grep -iE '\*.[ps]ga' |\
cut -d= -f2

It returned all kind of different values, like:

1500M
2g
754974720
2G
351272960
2g
2561671168

To sum them, one needs to convert everything to be of the same order. In this case values without a postfix were in kilobytes, so we should convert gigabytes and megabytes to kilobytes. Since I’d need a calculator later on anyway, I used ‘sed‘ to convert these values as shown below:

find /oracle/product/*/db_1/dbs/spfile*.ora |\
xargs strings -a |\
grep target |\
grep -iE '\*.[ps]ga' |\
cut -d= -f2 |\
sed -e 's/G/*1024*1024*1024/gi' |\
sed -e 's/M/*1024*1024/gi'

This results in:

1500*1024*1024
2*1024*1024*1024
754974720
2*1024*1024*1024
351272960
2*1024*1024*1024
2561671168

For this to be calculated with ‘bc‘, we need it all on one line and ‘+’ between the rows. That step can be done with the command ‘paste‘. We want all values to be on one line (-s) and use ‘+’ as a delimiter (-d):

find /oracle/product/*/db_1/dbs/spfile*.ora |\
xargs strings -a |\
grep target |\
grep -iE '\*.[ps]ga' |\
cut -d= -f2 |\
sed -e 's/G/*1024*1024*1024/gi' |\
sed -e 's/M/*1024*1024/gi' |\
paste -s -d+

This results in:

1500*1024*1024+2*1024*1024*1024+754974720+2*1024*1024*1024+351272960+2*1024*1024*1024+2561671168

Final step is to feed this calculation to ‘bc’ and you’re done:

find /oracle/product/*/db_1/dbs/spfile*.ora |\
xargs strings -a |\
grep target |\
grep -iE '\*.[ps]ga' |\
cut -d= -f2 |\
sed -e 's/G/*1024*1024*1024/gi' |\
sed -e 's/M/*1024*1024/gi' |\
paste -s -d+ |\
bc

This results in:

11683233792

If you want to convert it to display gigabytes instead of kilobytes, add this ‘awk‘ command to instruct ‘bc‘ to divide and display two decimals (otherwise it will floor down the value):

find /oracle/product/*/db_1/dbs/spfile*.ora |\
xargs strings -a |\
grep target |\
grep -iE '\*.[ps]ga' |\
cut -d= -f2 |\
sed -e 's/G/*1024*1024*1024/gi' |\
sed -e 's/M/*1024*1024/gi' |\
paste -s -d+ |\
awk {'print "scale=2; (" $1 ")/1024/1024/1024"'} |\
bc

This results in:

10.88

This means 10.88 GB of memory has been allocated to databases.

A nice example of the power of the Linux shell 🙂

Recently I was looking for a way to SSH from a network that blocked my outgoing SSH connection. I’d be nice to have a way around firewalls and be able to access your private Linux terminal. To be able to debug a problem from an remote location, for example.

A collegue suggested a tool called ‘Shell In A Box‘. Shell In A Box implements a web server that can export arbitrary command line tools to a web based terminal emulator using just JavaScript and CSS without any additional browser plugins. This means: connecting your browser via HTTPS to your own hosted Shell In A Box web site, and access a Linux terminal from there.

How cool is that? In this blog I’ll show you how to set it up in a secure way.

Building and installing Shell In A Box
I want to setup Shell In A Box on my Raspberry Pi. It’s a great device running Linux that has a very small energy consumption footprint. Ideal for an always-on device I’d say!

Since there is no package available, we’ve to compile our own. It’s best to get the sources from Github (original here), since the Github repository contains some patches and fixes for issues on Firefox.

These commands install the required dependencies, clone the Git repository and start building:

apt-get install git dpkg-dev debhelper autotools-dev libssl-dev libpam0g-dev zlib1g-dev libssl1.0.0 libpam0g openssl
git clone https://github.com/pythonanywhere/shellinabox_fork
cd shellinabox_fork
dpkg-buildpackage

During my first attempt, I ran into this problem:

dpkg-source -b shellinabox-2.14
dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../shellinabox_2.14.orig.tar.{bz2,gz,lzma,xz}
dpkg-buildpackage: error: dpkg-source -b shellinabox-2.14 gave error exit status 255

When grepping for ‘quilt’ I found a file called ‘/debian/source/format’. From what I can tell this does not do anything important, so I ended up deleting the file. Guess what, it now works.

rm ./debian/source/format

Build the package again, this should now succeed.

dpkg-buildpackage

This process will take some time (especially on the Raspberry Pi). Afterwards you’ll find the .deb file ready to be installed.

dpkg -i ../shellinabox_2.14-1_armhf.deb

I changed the configuration, to disallow the build-in SSL and to bind to localhost only. I did this because another web server will serve our terminal. I will explain in a minute.

vim /etc/default/shellinabox

And edit this line:

SHELLINABOX_ARGS="--no-beep -s /terminal:LOGIN --disable-ssl --localhost-only"

Finally, restart the deamon:

/etc/init.d/shellinabox restart

And check if all went well:

/etc/init.d/shellinabox status

You should see:

Shell In A Box Daemon is running

Another way to verify is to check the open ports:

netstat -ntl

You should see:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State 
tcp 0 0 127.0.0.1:4200 0.0.0.0:* LISTEN


Setting up Lighttpd as a proxy

Shell In A Box runs on port 4200 by default. Although this can be changed to a more common 80 or even 443, this is not what I want. I decided to integrate it with another webserver, to be able to combine other services and use just one url (and one SSL certificate). Since the Raspberry Pi isn’t that powerful, I choose Lighttpd.

apt-get instal lighttpd
cd /etc/lighttpd/conf-enabled
ln -s ../conf-available/10-proxy.conf

This installs Lighttpd and enables Proxy support. Now add the Proxy config:

vim /etc/lighttpd/lighttpd.conf

And add:

proxy.server = (
 "/terminal" =>
  ( (
    "host" => "127.0.0.1",
    "port" => 4200
  ) )
)

Save and restart Lighttpd:

/etc/init.d/lighttpd restart

Connect to http://pi.example.org/terminal and your Shell In A Box terminal should appear.

Although this is cool already, we’re not quite there. No one will SSH on an unencrypted web page, right? So, we’ll configure an SSL certificate to enable encryption. For double safety, we’ll also set a username/password on the web page. One then needs to know this password to access the login promt, and needs a valid local username/password to really use the terminal.

Adding encryption with SSL
By using a HTTPS-url, our traffic is encrypted. Let’s generate a private key (and remove the passphrase):

openssl genrsa -des3 -out pi.example.org.key 2048
cp -pr pi.example.org.key pi.example.org.key.passwd
openssl rsa -in pi.example.org.key.passwd -out pi.example.org.key

If you do not remove the passphrase, you will need to type it every time you start the web server. To request a SSL-certificate, you need to supply a CSR (Certificate Signing Request) and send that to a SSL provider such as Thawte or Verisign.

openssl req -new -key pi.example.org.key -out pi.example.org.csr

To be able to continue now, let’s self-sign the certificate:

openssl x509 -in pi.example.org.csr -out pi.example.org.pem -req -signkey pi.example.org.key -days 365
cat pi.example.org.key >> pi.example.org.pem

A self-signed certificate will display a warning in our browser, but that’s ok for now. Once the real certificate comes back from our SSL provider, it’s easy to replace it. The warning will then disappear.

Time to tell Lighttpd about our certificate:

vim /etc/lighttpd/lighttpd.conf

Add these lines:

$SERVER["socket"] == "10.0.0.10:443" {
  ssl.engine = "enable"
  ssl.pemfile = "/etc/lighttpd/ssl/pi.example.org/pi.example.org.pem"
  server.name = "pi.example.org"
  server.document-root = "/home/lighttpd/pi.example.org/https"
  server.errorlog = "/var/log/lighttpd/pi.example.org_serror.log"
  accesslog.filename = "/var/log/lighttpd/pi.example.org_saccess.log"
}

And restart Lighttpd:

/etc/init.d/lighttpd restart

Now Shell In A Box should be available on: https://pi.example.org/terminal

Enhancing security by adding HTTP-auth
Since the /terminal page now makes an actual terminal available to web users, I added an extra password for security. You can use the ‘HTTP Auth’ method for this. It will pop up a message box that requires an valid username/password before the /terminal page is shown.

First enable the module:

cd /etc/lighttpd/conf-enabled
ln -s ../conf-available/05-auth.conf

Then extend the config of the virtual host config you created above. The final result should be:

$SERVER["socket"] == "10.0.0.10:443" {
  ssl.engine = "enable"
  ssl.pemfile = "/etc/lighttpd/ssl/pi.example.org/pi.example.org.pem"

  server.name = "pi.example.org"
  server.document-root = "/home/lighttpd/pi.example.org/https"
  server.errorlog = "/var/log/lighttpd/pi.example.org_serror.log"
  accesslog.filename = "/var/log/lighttpd/pi.example.org_saccess.log"

  auth.debug = 2
  auth.backend = "htpasswd"
  auth.backend.htpasswd.userfile = "/etc/lighttpd/shellinabox-htpasswd"

  auth.require = ( "/terminal/" =>
    (
      "method" => "basic",
      "realm" => "Password protected area",
      "require" => "user=remibergsma"
    )
  )
}

Reload Lighttpd to make the changes active:

/etc/init.d/lighttpd reload

To set a password:

apt-get install apache2-utils
htpasswd -c -m /etc/lighttpd/shellinabox-htpasswd remibergsma

You can enter multiple users, just remember to remove the ‘-c’ flag when adding more users, as this overwrites the current file.

When you visit https://pi.example.org/terminal you will need to enter a valid username and password, before the page loads.

The final result: SSH in a browser window!
You should now be able to use a terminal via your own protected webpage. It’s mostly like a real terminal/SSH session but from a browser. Wow 🙂

Shell In A Box in action

Shell In A Box in action

 

I always use GNU Screen, so I know for sure my commands keep running whatever happens.

Using GNU Screen in a browser

Using GNU Screen in a browser

 

I’m using SSH’s key-based authentication to login to my servers using a SSH key pair. Of course my private key is protected with a password of its own. This has the consequence that even with key-based authentication, you still need to type a password: your private key passphrase. And you need to type it on every single connect. Although this is far more secure that having no passphrase on your private key, typing it so many times is a bit over done.

In this blog I’ll show you how to setup a system where you enter the passphrase only once per day when you start a new terminal session. For me this works great, as I now only type my passphrase in the morning when I login to my workstation and am able to connect password-less the rest of the day. Whenever I leave my desk, my screen is locked so no one can access my key.

Setting up key-based authentication

The first step is to generate a key pair to use.

ssh-keygen -t rsa -b 2048

The output is something like below. Enter your passphrase twice, when prompted.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/remi/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/remi/.ssh/id_rsa.
Your public key has been saved in /home/remi/.ssh/id_rsa.pub.
The key fingerprint is:
41:22:af:8b:4b:aa:94:34:a4:fa:5a:eb:91:4c:5b:5e remi@wks

The key's randomart image is:

+--[ RSA 2048]----+
|    . . .        |
|     o o         |
| .    . .        |
|o    .   .       |
|.o. o E S        |
|ooo* o           |
|.oO o            |
|.= +             |
|=o=              |
+-----------------+

To setup key-based authentication, copy your public key to the ‘.ssh/authorized_keys’ file on the server you want to connect to. You can do this by using ‘ssh-copy-id‘. You need to have access to the user account you want to connect to because you need to type its password to authenticate the copying.

ssh-copy-id -i ~/.ssh/id_rsa.pub user@server

You can now login to this server by typing only the passphrase of your private key, instead of typing the user account’s password.

Using ssh-agent to manage private keys
When you want to take this to the next level, you can use ‘ssh-agent‘ to manage your SSH keys. This program is started at login and remembers the keys you add each session. Through use of environment variables the agent can be located and automatically used for authentication when logging in to other machines using SSH.

The idea is that the agent is run in the user’s local PC, laptop, or terminal.  Authentication data need not be stored on any other machine, and authentication passphrases never go over the network. However, the connection to the agent is forwarded over SSH remote logins, and the user can thus use the privileges given by the identities anywhere in the network in a secure way.

To add a key, run ‘ssh-add‘ and type the passphrase of your private key. This is needed only once per session. After you’ve added your key to ‘ssh-agent‘, you should be able to SSH to the local machine without entering a password. This is more secure than a key pair without a passphrase because whenever your key gets lost, one still needs to type your passphrase to get access.

Use case
At work, I SSH to a central server from which I can access all the other servers. My public key is distributed to all servers, my private key is on my workstation. Now, for secure password-less logins to work, I need to be able to access my local ssh-agent from this central server. When you SSH to your server as I describe above, you don’t have to type the passphrase. When you have a look at the environment variables, you’ll notice SSH_AUTH_SOCK is set. This variable holds the path to the socket connecting to your ‘ssh-agent‘ running on your local machine. Authentication is now done this way!

Getting this to work with GNU screen
There’s a little problem when using this technique with GNU screen. When you reconnect to a running screen session from another location, the environment variable inside screen is not updated and so you need to type the password for your private key on each connect again. I found this post that describes a solution, which I further enhanced to make it work correctly for me:

export SSH_AUTH_SOCK=$(find /tmp/ssh-* -user `whoami` -name agent\* -printf '%T@ %p\n' 2>/dev/null | sort -k 1nr | sed 's/^[^ ]* //' | head -n 1)

This one-liner looks for the most recent ssh-agent socket, owned by the current user and sets the SSH_AUTH_SOCK environment variable to this socket. You can now use this socket again to authenticate to your locally running ‘ssh-agent‘.

To make this a little easier, I added this scrip ‘update-ssh-agent-socket.sh‘ on my home directory on the central server (also available on github):

#!/bin/bash

echo "Updating ssh-agent socket environment.."
echo "Current value:  $SSH_AUTH_SOCK"

export SSH_AUTH_SOCK=$(find /tmp/ssh-* -user `whoami` -name agent\* -printf '%T@ %p\n' 2>/dev/null | sort -k 1nr | sed 's/^[^ ]* //' | head -n 1)

if [ $? -gt 0 ]; then
 echo "ERROR!"
 exit $?
fi

echo "New value:  $SSH_AUTH_SOCK"
echo "All done!"

Run it like this to use the most recent ssh-agent socket (note the dot at the start):

. ~/update-ssh-agent-socket.sh

Usually I type ‘<CTR>+r update <enter>’ to quickly run this command from history. This is the quickest way I could think of to get everything working and make it easy to quickly login to our servers.

Changes to production systems should be tested on a development system and then be deployed using configuration management (such as Puppet), if you ask me. Sometimes I first run commands by hand (on a test system) to find out the right ones, the right order, etc. In those cases I find it useful to automatically document what I type and be able to ‘replay’ it later on.

Why? Because when I write the Puppet configuration needed to deploy the change, I want to be sure all manual commands I ran make it to the Puppet manifest. Being able to replay what I did allows me to do so with ease.

Here’s how to record your key strokes using the ‘script’ Linux utility:

script change1234.script -t 2> change1234.timing

The ‘script’ utility logs all commands you enter to the file ‘change1234.script’. Furthermore it saves the timing data to a file called ‘change1234.timing’. Beware that everything is saved, including errors and typo’s.

The timing data allows to replay the script using the same timing as when the session was recorded. It gives a good representation of what happened. To replay, simply run:

scriptreplay change1234.timing change1234.script

You can replay it as many times as you like.

The .script and .timing files are just plain text files. This means you can ‘grep’ them to quickly find commands. For example, to display the ‘sed’ commands you used:

grep sed change1234.script

As you can see, saving (and documenting) your commands is easy and offers some nice features. It even allows you to forget what exactly you did… 😉

Sometimes files may be filled up with null characters that look like ^@ when you open them in a text editor. This may happen when a disk becomes full, or when you rename a logfile while an application is still writing to it.

I ran into this problem today, and I fixed it using a command called ‘tr’. This is a utility capable of translating or deleting characters from standard input/output. It means you can use it to ‘pipe’ input to it, and send the output to a new file. For example:

cat file.log | tr -d '\000'  > new_file.log 

Note: when using this in a script, you might need to escape that backslash.

What does this command do? Using the -d switch we delete a character. A backslash followed by three 0’s represents the null character. This just deletes these characters and writes the result to a new file. Problem solved!