Now that I’m using CFEngine for some time, I’m exploring more and more possibilities. CFEngine is in fact a replacement of a big part of our Zenoss monitoring system. Since CFEngine does not only notice problems, but usually also fixes them, this makes perfect sense.
Recently I created a promise that monitors the disk space on our servers. Since we use Logstash to monitor our logs, all CFEngine needs to do is log a warning and our team will have a look. I will write about Logstash another time 😉
To monitor disk space, I use two bundles: one that is included from ‘promises.cf’ and a second one that is called from the first one.
The ‘diskspace’ bundle looks like this:
bundle agent diskspace { vars: "disks[root][filesystem]" string => "/"; "disks[root][minfree]" string => "500M"; "disks[root][handle]" string => "system_root_fs_check"; "disks[root][comment]" string => "/ filesystem check"; "disks[root][class]" string => "system_root_full"; "disks[root][expire]" string => "60"; "disks[var][filesystem]" string => "/var"; "disks[var][minfree]" string => "500M"; "disks[var][handle]" string => "var_fs_check"; "disks[var][comment]" string => "/var filesystem check"; "disks[var][class]" string => "var_full"; "disks[var][expire]" string => "60"; apache_webserver:: "disks[webdata][filesystem]" string => "/webdata"; "disks[webdata][minfree]" string => "1G"; "disks[webdata][handle]" string => "webdata_fs_check"; "disks[webdata][comment]" string => "/webdata filesystem check"; "disks[webdata][class]" string => "webdata_full"; "disks[webdata][expire]" string => "60"; someserver01:: "disks[tmp][filesystem]" string => "/tmp"; "disks[tmp][minfree]" string => "1G"; "disks[tmp][handle]" string => "tmp_fs_check"; "disks[tmp][comment]" string => "/tmp filesystem check"; "disks[tmp][class]" string => "tmp_full"; "disks[tmp][expire]" string => "30"; methods: "disks" usebundle => checkdisk("diskspace.disks"); }
All it does is define the configuration, depending on the classes that are set. Some are disks monitored on all servers (like ‘/’ and ‘/var’ in this example) and some are monitored only when the specified class is set (like ‘apache_webserver’ and ‘someserver01’). These are classes I defined in ‘promises.cf’ based on several custom criteria.
The ‘checkdisk’ bundle does the real job:
bundle agent checkdisk(d) { vars: "disk" slist => getindices("$(d)"); storage: "$($(d)[$(disk)][filesystem])" handle => "$($(d)[$(disk)][handle])", comment => "$($(d)[$(disk)][comment])", action => if_elapsed("$($(d)[$(disk)][expire])"), classes => if_notkept("$($(d)[$(disk)][class])"), volume => min_free_space("$($(d)[$(disk)][minfree])"); }
Although this may look cryptic, it is a nice and abstract way to organize the promise. This is ‘implicit’ looping in action: if the ‘disk’ variable contains more than one item, CFEngine will automatically process the code for each item. In this case, this means a promise is created for all the disks specified.
If you prefer a percentage, instead of fixed free size, you could use ‘freespace’ instead of ‘min_free_space’ in the ‘checkdisk’ bundle.
When a disk is below threshold, this log message is written:
2013-11-01T17:35:42+0100 error: /diskspace/methods/'disks'/checkdisk/storage/'$($(d)[$(disk)][filesystem])': Disk space under 2097152 kB for volume containing '/' (802480 kB free) 2013-11-01T17:35:42+0100 error: /diskspace/methods/'disks'/checkdisk/storage/'$($(d)[$(disk)][filesystem])': Disk space under 2097152 kB for volume containing '/tmp' (1887956 kB free)
Apart from reporting, you could even instruct CFEngine to clean up certain files when disk space becomes low or to run a script. You would then use the specified ‘class’ that is set when the disk has low free disk space (‘tmp_full’ for example). Anything is possible!
Very Cool!
Glad you like it 🙂