Archives For 31 December 2013
I’m currently finalizing the CFEngine 3 setup at my $current_work because by the end of the month I will start a new job. In a little over a year, I fully automated the Linux sysadmin team. From now on, only 2 sysadmins are needed to keep everything running. Since almost everything is automated using CFEngine 3, it’s very important that CFEngine is running at all times so it can keep an eye on the systems and thus prevent problems from happening.
I’ve developed an init script, that makes sure CFEngine is installed and bootstrapped to the production CFEngine policy server. This init script is added in the post-install phase of the automatic installation. This gets everything started and from there on CFEngine kicks in and takes control. That same init script is also maintained with CFEngine. This is done so it cannot easily be removed or disabled.
Also, when CFEngine is not running (anymore) it should be restarted. A cron job is setup to do this. This cron job is also setup using CFEngine. It is using regular cron on the OS, of course. If all else fails, this cron job can also install CFEngine in the event it might be removed. Last thing it does, is automatically recover from ‘SIGPIPE’ bug we sometimes encounter on SLES 11.
To summarize:
– an init script (runs every boot) makes sure CFEngine is installed and bootstrapped
– a hourly cron job makes sure the CFEngine daemons are actually running
– CFEngine itself ensures both the cron job and init script are properly configured
This makes it a bit harder to (accidentally) remove CFEngine, don’t you think?!
Reporting servers that do not talk to the Policy server anymore
Now, imagine someone figures a way to disable CFEngine anyway. How would we know? The CFEngine Policy server can report this using a promise. It reports it via syslog, so it will show up in Logstash. The bundle looks like this:
bundle agent notseenreport
{
classes:
"display_report" expression => "Hr08.Min00_05";
vars:
# Default to empty list
"myhosts" slist => { };
display_report::
"myhosts" slist => { hostsseen("24","notseen","name") };
reports:
"CFHub-Production: Did not talk to $(myhosts) for over 24 hours now";
}
We’ve set this up on both Production and Pre-Production Policy servers.
How to temporary disable CFEngine?
On the other side, sometimes you want to temporary disable CFEngine. For example to debug an issue on a test server. After a discussion in our team, we came up with an easy solution: when a so-called ‘Do Not Run‘ file exists on a server, we should instruct CFEngine to do nothing. We use the file ‘/etc/CFEngine_donotrun‘ for this, so you’d need ‘root‘ privileges or equal to use it.
In ‘promise.cf‘ a class is set when the file is found:
"donotrun" expression => fileexists("/etc/CFEngine_donotrun");
For our setup we’re using a method detailed in ‘A CFEngine Case Study‘. We added the new class:
!donotrun::
"sequence" slist => getindices("bundles");
"inputs" slist => getvalues("bundles");
donotrun::
"sequence" slist => {};
"inputs" slist => {};
reports:
donotrun::
"The 'DoNotRun' file was found at /etc/CFEngine_donotrun, exiting.";
In other words, when the ‘Do Not Run‘ file is found, this is reported to syslog and no bundles are included for execution: CFEngine then does effectively nothing.
An overview of servers that have a ‘Do Not Run‘ file appears in our Logstash dashboard. This makes them visible and we look into then on a regular basis. It’s good practice to put the reason why in the ‘Do Not Run‘ file, so you know why it was disabled and when. Of course, this should only be used for a small period of time.
Making sure CFEngine runs at all times makes your setup more robust, because CFEngine fixes a lot of common problems that might occur. On the other hand, having an easy way to temporary disable CFEngine also prevents all kind of hacks to ‘get rid of it’ while debugging stuff (and then forgetting to turn it back on). I’ve found this approach to work pretty good.
Update:
After publishing this post, I got some nice feedback. Both Nick Anderson (in the comments) and Brian Bennett (via twitter) pointed me into the direction of CFEngine’s so called ‘abortclasses‘ feature. The documentation can be found on the CFEngine site. To implement it, you need to add the following to a ‘body agent control‘ statement. There’s one defined in ‘cf_agent.cf‘, so you could simply add:
abortclasses => { "donotrun" };
Another nice thing to note, is that others have also implemented similar solutions. Mitch Lewandowski told me via twitter he uses a filed simply called ‘/nocf‘ for this purpose and Nick Anderson (in the comments) came up with an even funnier name: ‘/COWBOY‘.
Thanks for all the nice feedback! 🙂
Recently a colleague came by and asked whether I could help him with finding out how much memory was already assigned to Oracle databases. It was fun to find out what was exactly he needed and to construct the one-liner bit by bit, as I detail below. When it seemed to work ok, I wrapped it in a small shell script to make it more flexible and reusable and deployed it to all Oracle database servers using CFEngine. You can find the final shell script on Github. I’d suggest you use the shell script instead of the one-liners below, as it is more accurate.
But let’s now play a bit with the shell. Since part of the memory used by Oracle is dynamically assigned, using tools like ‘free‘ will not really help to determine how much memory is allocated. My colleague told me we’d to look at the ‘PGA‘ (Program Global Area) and ‘SGA‘ (System Global Area) values in the Oracle settings of each database.
From the Oracle docs:
The SGA is a group of shared memory structures, known as SGA components, that contain data and control information for one Oracle Database instance. The SGA is shared by all server and background processes. Examples of data stored in the SGA include cached data blocks and shared SQL areas. A PGA is a memory region that contains data and control information for a server process. It is nonshared memory created by Oracle Database when a server process is started. Access to the PGA is exclusive to the server process. There is one PGA for each server process. Background processes also allocate their own PGAs. The total memory used by all individual PGAs is known as the total instance PGA memory, and the collection of individual PGAs is referred to as the total instance PGA, or just instance PGA. You use database initialization parameters to set the size of the instance PGA, not individual PGAs.
These settings can be found in the ‘spfile*.ora‘ of a given database. This is how I found all of them:
find /oracle/product/*/db_1/dbs/spfile*.ora
Because these are binary data files, so you need ‘strings‘ to get the readable out put. The settings we were looking for had ‘target‘ in their name, so we could grep them out like this:
strings -a /oracle/product/11.x.y.z/db_1/dbs/spfiledbname.ora |\ grep -i target
We’re especially interested in settings that started with ‘*.pga‘ or ‘*.sga‘. Example:
*.pga_aggregate_target=955252736 *.sga_target=1610612736
Of course, grep can do that too. On the lines we found, the value of the setting (next to the ‘=’) can be in kilobytes (as shown above), but also in megabytes (ending in ‘M’ or ‘m’) or gigabytes (ending in ‘G’ or ‘g’).
The question was: How to quickly sum these values on the command line to get an impression of the assigned memory. My colleagues usually did this by hand, but now there were a bit too many. You shouldn’t do this by hand anyway, if you ask me. So I gave it a try:
First we need to select the correct lines of all ‘spfiles‘ of all databases on a server:
find /oracle/product/*/db_1/dbs/spfile*.ora |\ xargs strings -a |\ grep target |\ grep -iE '\*.[ps]ga'
This resulted a list of all the settings we were looking for. The same as above, but for all databases, one per row. The next thing to do, to be able to sum them up, is to remove everything except the values we want to sum. The command ‘cut’ can do this, using a delimeter ‘=’ and then the second field. It looks like this:
find /oracle/product/*/db_1/dbs/spfile*.ora |\ xargs strings -a |\ grep target |\ grep -iE '\*.[ps]ga' |\ cut -d= -f2
It returned all kind of different values, like:
1500M 2g 754974720 2G 351272960 2g 2561671168
To sum them, one needs to convert everything to be of the same order. In this case values without a postfix were in kilobytes, so we should convert gigabytes and megabytes to kilobytes. Since I’d need a calculator later on anyway, I used ‘sed‘ to convert these values as shown below:
find /oracle/product/*/db_1/dbs/spfile*.ora |\ xargs strings -a |\ grep target |\ grep -iE '\*.[ps]ga' |\ cut -d= -f2 |\ sed -e 's/G/*1024*1024*1024/gi' |\ sed -e 's/M/*1024*1024/gi'
This results in:
1500*1024*1024 2*1024*1024*1024 754974720 2*1024*1024*1024 351272960 2*1024*1024*1024 2561671168
For this to be calculated with ‘bc‘, we need it all on one line and ‘+’ between the rows. That step can be done with the command ‘paste‘. We want all values to be on one line (-s) and use ‘+’ as a delimiter (-d):
find /oracle/product/*/db_1/dbs/spfile*.ora |\ xargs strings -a |\ grep target |\ grep -iE '\*.[ps]ga' |\ cut -d= -f2 |\ sed -e 's/G/*1024*1024*1024/gi' |\ sed -e 's/M/*1024*1024/gi' |\ paste -s -d+
This results in:
1500*1024*1024+2*1024*1024*1024+754974720+2*1024*1024*1024+351272960+2*1024*1024*1024+2561671168
Final step is to feed this calculation to ‘bc’ and you’re done:
find /oracle/product/*/db_1/dbs/spfile*.ora |\ xargs strings -a |\ grep target |\ grep -iE '\*.[ps]ga' |\ cut -d= -f2 |\ sed -e 's/G/*1024*1024*1024/gi' |\ sed -e 's/M/*1024*1024/gi' |\ paste -s -d+ |\ bc
This results in:
11683233792
If you want to convert it to display gigabytes instead of kilobytes, add this ‘awk‘ command to instruct ‘bc‘ to divide and display two decimals (otherwise it will floor down the value):
find /oracle/product/*/db_1/dbs/spfile*.ora |\
xargs strings -a |\
grep target |\
grep -iE '\*.[ps]ga' |\
cut -d= -f2 |\
sed -e 's/G/*1024*1024*1024/gi' |\
sed -e 's/M/*1024*1024/gi' |\
paste -s -d+ |\
awk {'print "scale=2; (" $1 ")/1024/1024/1024"'} |\
bc
This results in:
10.88
This means 10.88 GB of memory has been allocated to databases.
A nice example of the power of the Linux shell 🙂
Remi Bergsma's blog 