Chef Gets Push in Q1/2014

Sysadvent features a puppetlabs sponsered article (yes, honestly, check the bottom of the page!) about chef enterprise getting push support. It is supposed to be included in the open source release in Q1/2014.

With this change you can use a push jobs cookbook to define jobs and an extended "knife" with new commands to start and query about jobs:

knife job start ...
knife job list


knife node status ...

will tell about job execution status on the remote node.

At a first glance it seems nice. Then again I feel worried when this is intended to get rid of SSH keys. Why do we need to get rid of them exactly? And in exchange for what?

Simple Chef to Nagios Hostgroup Export

When you are automatizing with chef and use plain Nagios for monitoring you will find duplication quite some configuration. One large part is the hostgroup definitions which usually map many of the chef roles. So if the roles are defined in chef anyway they should be sync'ed to Nagios.

Using "knife" one can extract the roles of a node like this

knife node show -a roles $node | grep -v "^roles:"

Scripting The Role Dumping

Note though that knife only shows roles that were applied on the server already. But this shouldn't be a big problem for a synchronization solution. Next step is to create a usable hostgroup definition in Nagios. To avoid colliding with existing hostgroups let's prefix the generated hostgroup names with "chef-". The only challenge is the regrouping of the role lists given per node by chef into host name lists per role. In Bash 4 using an fancy hash this could be done like this:

declare -A roles

for node in $(knife node list); do
   for role in $(knife node show -a roles $i |grep -v "roles" ); do
      roles["$role"]=${roles["$role"]}"$i "

Given this it is easy to dump Icinga hostgroup definitions. For example

for role in ${!roles[*]}; do
   echo "define hostgroup {
   hostgroup_name chef-$role
   members ${roles[$role]}

That makes ~15 lines of shell script and a cronjob entry to integrate Chef with Nagios. Of course you also need to ensure that each host name provided by chef has a Nagios host definition. If you know how it resolves you could just dump a host definition while looping over the host list. In any case there is no excuse not to export the chef config :-)

Easy Migrating

Migrating to such an export is easy by using the "chef-" namespace prefix for generated hostgroups. This allows you to smoothly migrate existing Nagions definitions at your own pace. Be sure to only reload Nagios and not restart via cron and to do it at reasonable time to avoid breaking things.

What Job Offers Tell About Architectures #4

Part 4 of the Unix/Linux system administrator job offer and technologies survey. Here are several new companies. Note that some positions are actually developer positions. I chose those because they still told much about the software stack used. Check the "[src]" for the original reference, but know that it might be already offline.

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
Brose [src] AIX, Linux, Windows HA ABAP Oracle, MaxDB SAP - - - -
DATEV eG [src] - Java, Javascript DB2 J2EE, DOJO, JUnit, Mockito - - - -
DATEV eG [src] z/OS Cobol, Java, Javascript DB2, VSAM - - - - -
Elektrobit Automotive GmbH [src] Unix Java Oracle Tomcat, WebSphere, Spring, Eclipse RCP - - - -
Actano GmbH [src] Win, Mac OSX, Linux - MSSQL, Oracle, MySQL Firewall, Exchange - Nagios, Centreon vSphere, HyperV, AWS, Profitbricks -
Ratbacher [src] AIX, Linux - DB2, Oracle Tivoli Storage Manager IBM Power - - -
United Experts [src] Debian Lenny, Squeeze, Wheezy - MySQL, Postgresql Haproxy, Apache, Tomcat Network HW, Blades, Storages Nagios, Munin, Icinga, Zabbix Xen, KVM -
Texas Instruments [src] Solaris, Linux Shell, Python, Perl Oracle - x86 and SPARC HW components, Blades, SAN, NAS - - -
Media Saturn [src] Unix - - Tomcat, Citrix, Cluster, ITIL - - - -
easyCash [src] Linux, AIX Shell - Apache, Tomcat, JBoss, Weblogic, ITIL - Nagios - -

The Interesting Things

Nothing unexpected really. Well, except noone seems to use automation. All offers look very serious, as non-startup-like businesses should! Among them several AIX users and even a z/OS position offered by German tax services provider DATEV.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

What Job Offers Tell About Architectures #3

Part 3 of the Unix/Linux system administrator job offer and technologies survey. Here are 8 new companies:

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
Sunbeam GmbH [src] Debian, CentOS PHP MySQL Typo3, Piwik, Mantis, OpenX - - OpenVZ -
Air Berlin [src] Windows - Oracle 11g, MS SQL 2008 - - - - -
BearingPoint [src] Unix, Linux Shell, Java Oracle Apache, Tomcat - - - -
COC AG [src] Unix, Linux, Windows Server - ADS - - - VMWare, MS, Citrix -
dm Pharmacy [src] RHEL, Window Server 2008 Shell, Ruby, Java Varnish, Apache, JBoss, Tomcat - - - VMWare Chef or Puppet
Groupon GmbH [src] Linux - MySQL, Postgres Apache, Tomcat, Hudson, Jenkins, LDAP - - vSphere, VirtualBox cfengine2/3, Puppet
HUK-Coburg Insurance [src] - - DB2 DWH - - - -
InVision AG [src] Linux - Postgres, MySQL, Redis - - - - Chef, Puppet, Capistrano

The Interesting Things

Well none of the above companies needs monitoring :-)

And GroupOn is one of the few companies so far asking for cfengine. And for me the first time I heard about capistrano (requested by Invision AG). The github repository tells it is a:

"Remote multi-server automation tool"

implemented in Ruby modelled from the Rake DSL. I'm worried that I missed this so long as it seems to be quite popular on github.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

PHP ini_set() Examples

Syntax of ini_set()

The ini_set() syntax is simple:

string ini_set ( string $varname , string $newvalue )

it is just a key value setter. The important question is which values can be set. Below you find a list of working examples. Please note that you cannot change all php.ini options especially those that need to be set before PHP initializes.

Useful Working ini_set() Examples

1. Enabling error display

On production sites you typically do not show errors in the page for usability and security reasons. But when you when you debug something live you might want to enable it temporarily and just for you:

# Assuming is your IP...
if ( $_SERVER["REMOTE_ADDR"] == "") {
    ini_set('display_errors', '1');

Note: you may want to combine this with

error_reporting(E_ALL | E_STRICT);

2. Changing Memory Limit

When you need to increase memory from within the code:


Note though that this might be prevent by a Suhosin hardended PHP installation.

3. Adding include paths

Normally this shouldn't be necessary. It is way cleaner to do it in php.ini, but if you bundle libraries and you administrator doesn't know:

<?php ini_set('include_path',ini_get('include_path').':../my-libs:');  ?>

When You Cannot Use ini_set()

For most php.ini settings you can't use ini_set(). To workaround consider deploying a .htaccess along with your code as this .htaccess can provide all PHP options to overwrite the default php.ini settings.

For example to change the HTTP POST limit add this line to a .htaccess read by your webserver:

php_value post_max_size 2500000

Note how the "php_value" prefix indicates settings for PHP. So simple syntax is

php_value <key name> <value>

What Job Offers Tell About Architectures #2

It is time for another round of having a look which technologies companies announce with job openings for system administrators.

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
1&1 Shared Hosting [src] Debian Perl, Bash - Apache, DRBD, Debian Packaging - - - -
1&1 Virtualisation [src] Linux Perl, Python - - NAS, SAN, iSCSI, NFS - VMWare, Xen, KVM, OpenStack, OpenNebula -
1&1 Monitoring [src] Linux, Windows Server Java, C/C++, Perl, Bash, Ruby - - - - - -
1&1 [src] Linux Perl, Bash, Python MySQL, Oracle, NoSQL Apache, Tomcat, Java - - - -
1&1 [src] Linux Java - Apache, Tomcat, JBoss - - - -
1&1 [src] Linux Perl, Bash, Python MySQL, Oracle, NoSQL, Cassandra - - - - Puppet [src] - - MySQL, MS SQL DRBD, heartbeat - - - -
First Colo [src] Linux Bash, PHP, Perl, Python Postgres, MySQL - Cisco, Juniper - - -
IKB Bank [src] SLES 10/11/12, Solaris 8/10 Bash, Perl Postgres, MySQL - Fibre Channel SAN, Storages: EMC, HP, IBM - - -
s'Oliver [src] AIX - - SAP, Tivoli TSM IBM SAN - PowerVM -

The Interesting Things

Well as you can see a lot of 1&1 offers. With 1&1 being on of the large successful German ISPs running the leading mail portals GMX and it is interesting to see their technology requirements. They seem to be focussing on Java, MySQL and Oracle with NoSQL and Apache Cassandra being the youngest startup-ish tool used. So for me 1&1 makes a somewhat old-school impression, which of course is not necessarily bad. As they indicate no standard monitoring solution I guess they use a proprietory or commercial solution.

What I found noteworthy about the other companies is the usage of DRBD. It's more commonly used than I'd expect so far.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

Large Website Technology Changes in October 2013

Last month I did a first indexing of mostly request header, HTML and DNS based information about the top 200 sites listed by Alexa and the top 100 German websites. All the information is freely available and extracted from the website responses.

The detailed results can be found here:

What Changed In A Month?

Now roughly a month later I repeated the scan to see what changes do happen and what trends might be noticable:

DNS-Prefetching THE HTML header based DNS prefetching is spreading. Two of the large adult content sites added it and Chinese search engine introduced it too. As in my experience it provides a small but measurable latency improvement on many sites I'd guess usage will further spread...

Added by,,,
Removed by

IPv6 IPv6 support did not spread in the last month. Just two sides that dropped their AAAA records.

Removed by,

Varnish One of the most significant changes is more and more sites announcing the usage of Varnish as a cache. This is sometimes combined with a CDN, sometimes without.

Introduced by,,,

Hiding Server Version Two more websites are now hiding their Apache version.

Removed version now:,

XSS Header XSS headers are not widely spread. No real change in adoption.

Added by
Removed by


All the results listed above are based on a simple scanning script. The results present a snapshot of the websites and a single response only. This is of course not necessarily an indicating for what techniques the site uses in daily operations!

Learning from Job Offers

Every other day you get one, fly over it and delete it. In a way each time it says exactly the same. Or does it?

Surely the different companies you read job offers from are using different technologies. Actually when announcing the position they are at their weakest time. They have to admit which technologies they use, how heterogenous they are and sometimes how old-style they are.

I think I'll from time to time compile some positions available online (not offers I got!) of mostly German and maybe more Berlin located companies and add them to an ever growing list "What Job Offers Tell". Below you find the first 10 company offers with their data.

Todays List

Company Link OS Languages Databases Software Hardware Monitoring Hosting Automation
Fraunhofer HHI [src] Ubuntu, Redhat, Debian C# - Windows AD, Windows Terminal Server, DFS Fibre Channel - vSphere 5 - [src] Debian, Redhat - - - - Icinga, Cacti, NewRelic - - [src] Debian - NoSQL, MongoDB LAMP - Nagios, Zookeeper, Corosync KVM -
KPMG [src] Windows Server 2003/2008 - MS SQL 2005/2008, SQL BI MS IIS - - - - [src] Linux Node.js MongoDB nginx, Varnish, Elastic Search - NewRelic - Puppet, SaltStack [src] Redhat PHP MySQL, Postgres, Redis, CouchDB Tomcat, JBoss, Apache, nginx - - Cloud Puppet, Foreman, Chef, Rex
Springer Online [src] CentOS, Redhat Bash, Python, Ruby MySQL, Postgres, MongoDB Apache, HaProxy, JBoss, Tomcat, Nginx, Varnish F5, Cisco Icinga, Graphite KVM, VMWare, Xen, AWS chef, Puppet
Teufel [src] Windows Powershell, Bash Exchange - - - VMWare, Hyper V chef, Puppet
T Systems [src] Linux Shell, Perl, PHP MySQL, Oracle heartbeat, DRBD, Apache, Tomcat, JBoss, Weblogic - - - -
Zalando [src] Windows - - MS ADS SAN - - -

The Interesting Parts

It gets interesting where positions go into details like with T-Systems using heartbeat and DRBD or Springer mentioning the real-time graphing engine Graphite. I also like maxdome using four different automation tools: Puppet, Foreman, Chef and Rex. That's two to many. A more exotic thing is using Apache Zookeeper, or is it more common than I think?

On the other hand some other candidates either have no automation needs, or the lack of hints indicates self-made automation or none at all.

Stay tuned for the next set of companies!

HowTo: Munin and rrdcached on Ubuntu 12.04

Let's expect you already have Munin installed and working and you want to reduce disk I/O and improve responsiveness by adding rrdcached... Here are the complete steps to integrate rrdcached:

Basic Installation

First install the stock package

apt-get install rrdcached

and integrate it with Munin:

  1. Enable the rrdcached socket line in /etc/munin/munin.conf
  2. Disable munin-html and munin-graph calls in /usr/bin/munin-cron
  3. Create /usr/bin/munin-graph with
    nice /usr/share/munin/munin-html $@ || exit 1
    nice /usr/share/munin/munin-graph --cron $@ || exit 1 

    and make it executable

  4. Add a cron job (e.g. to /etc/cron.d/munin) to start munin-graph:
    10 * * * *      munin if [ -x /usr/bin/munin-graph ]; then /usr/bin/munin-graph; fi

The Critical Stuff

To get Munin to use rrdcached on Ubuntu 12.04 ensure to follow these vital steps:

  1. Add "-s <webserver group>" to $OPT in /etc/init.d/rrdcached (in front of the first -l switch)
  2. Change "-b /var/lib/rrdcached/db/" to "-b /var/lib/munin" (or wherever you keep your RRDs)

So a patched default Debian/Ubuntu with Apache /etc/init.d/rrdcached would have

OPTS="-s www-data -l unix:/var/run/rrdcached.sock"
OPTS="$OPTS -j /var/lib/rrdcached/journal/ -F"
OPTS="$OPTS -b /var/lib/munin/ -B"

If you do not set the socket user with "-s" you will see "Permission denied" in /var/log/munin/munin-cgi-graph.log

[RRD ERROR] Unable to graph /var/lib/munin/
cgi-tmp/munin-cgi-graph/[...].png : Unable to connect to rrdcached: 
Permission denied

If you do not change the rrdcached working directory you will see "rrdc_flush" errors in your /var/log/munin/munin-cgi-graph.log

[RRD ERROR] Unable to graph /var/lib/munin/
cgi-tmp/munin-cgi-graph/[...].png : 
rrdc_flush (/var/lib/munin/[...].rrd) failed with status -1.

Some details on this can be found in the Munin wiki.

Liferea Code Repo Moved to github

I moved the source repo away from SourceForge away to GitHub.
It is currently located here:

If in doubt always follow the "Code" link from the website to find the repo.

Sorry, if this causes troubles for you. I'll contact all with current git write
access directly to see how we can continue on github and who will be able
to merge.

Please keep contributing! I think with github this can actually become
easier and more developers are familiar with its best practices.

Syndicate content Syndicate content