Large Website Technology Changes in November 2013

As in the last two months I performed another a indexing of the information reported by major websites. It covers mostly request header, HTML and DNS based information of the top 200 sites listed by Alexa and the top 100 German websites. All the information is freely available and only extracted from the website responses!

The detailed results can be found here:

What Changed?

DNS-Prefetching The HTML header based DNS prefetching is still there and gained another site: German job portal stepstone.de
IPv6 IPv6 support did not spread in the last month. Several sites have flapping visibility of their AAAA records.
Version Upgrades adf.ly upgrades from PHP 5.3.8 to 5.4.21.
rtl.de upgrades from Apache 2.2.21 to 2.4.6.
Webserver Change German health portal imedo.de switched from Mongrel to Apache
Hiding Server Version Porn site xnxx.com stopped reporting usage of PHP 5.3.6
CDN In 11/2013 conduit.com switched from Cotendo to Akamai CDN.
Hoster German couples portal parship.de seems to have changed hoster and is using BigIP F5 load balancers now.

Note: the website links lead to a history page for the measurements.

Caution!

All the results listed above are based on a simple scanning script. The results present a snapshot of the websites and a single response only. This is of course not necessarily an indicating for what techniques the site uses in daily operations!

GTK Tray StatusIcon Example with PyGI

Here is an example on how to build a GtkStatusIcon using PyGI (Python GObject). The code actually implements a libpeas plugin that could be used with any GTK+ project that allows GI plugins. The tray icon could respond to left clicking by toggling the application window like many instant messengers do. On right clicks it presents a menu with the options to toggle the application window or quit the application.

from gi.repository import GObject, Peas, PeasGtk, Gtk

class TrayiconPlugin (GObject.Object, PeasActivatable):
    __gtype_name__ = 'TrayiconPlugin'

    object = GObject.property (type=GObject.Object)

    def do_activate (self):
        self.staticon = Gtk.StatusIcon ()
	self.staticon.set_from_stock (Gtk.STOCK_ABOUT)
        self.staticon.connect ("activate", self.trayicon_activate)
        self.staticon.connect ("popup_menu", self.trayicon_popup)
        self.staticon.set_visible (True)

    def trayicon_activate (self, widget, data = None):
        print "toggle app window!"

    def trayicon_quit (self, widget, data = None):
        print "quit app!"

    def trayicon_popup (self, widget, button, time, data = None):
        self.menu = Gtk.Menu ()

        menuitem_toggle = Gtk.MenuItem ("Show / Hide")
        menuitem_quit = Gtk.MenuItem ("Quit")

        menuitem_toggle.connect ("activate", self.trayicon_activate)
        menuitem_quit.connect ("activate", self.trayicon_quit)

        self.menu.append (menuitem_toggle)
        self.menu.append (menuitem_quit)

        self.menu.show_all ()
	self.menu.popup(None, None, lambda w,x: self.staticon.position_menu(self.menu, self.staticon), self.staticon, 3, time)

    def do_deactivate (self):
        self.staticon.set_visible (False)
        del self.staticon

Website Technology Crawl 12/2013

I've update the What Large Sites Use page with the results crawled in December 2013. Check it out to see who is using what! New column for HTML DOCTYPE was added. I'll soon post the followup on the changes compared to November.

What Job Offers Tell About Architectures #5

Part 5 of the Unix/Linux system administrator job offer and technologies survey. Here are several new companies. This time watch out for "or's" in the table and the order of the alternative which might give an indication what the employer uses and what knowledge he deems equivalent. Also keep in mind that such "or's" indicate once more that each position might describe only wishes, but not that actual architecture the company uses. It depends on how open they want to be...

If you want to check for yourself click the "[src]" for the original reference, but know that it might be already offline.

Todays List

Company Sector Reference OS Languages Databases Software Hardware Monitoring Virtualization Automation
allesklar meinestadt.de Internet: Web [src] Linux MySQL, Postgres, Cassandra DB HA and Loadbalancing
Xing Social [src] Java, Scala ElasticSearch or (Lucene or Solr) Haadop, HBase, Cassandra
Hanwha Q Cells Industry [src] AIX System P, Storages, Tape
buch.de Online Store: Books [src] Redhat Apache, Tomcat, Redhat Satellite SAN yes
goodgame Studios Gaming [src] Debian, Redhat Shell MySQL Apache, Postfix Nagios, Munin, Icinga
International Algorithmic Trading GmbH Finance [src] Redhat, CentOS, Scientific Linux Bash, Perl yes
Lidl Online Store: Food [src] Redhat Shell MySQL VMWare
EOS UPTRADE GmbH Internet [src] Debian, CentOS PHP MySQL Cisco FW, Watchguard Icinga XenServer, VMWare Cloud Puppet

The Interesting Things

Well this is the first time I see someone using Scientific Linux. As with the previous posts automation doesn't seem to play a big role as only 2 out of 8 mentioned it. Also no startupish devops style offers here as the scripting languages seem to indicate. It seems to be just good old robust Unix administration :-)

And look at Q Cells, no wonder they went bankrupt using IBM hardware...

Note: You can find the complete index of all companies I checked so far here:
What Job Offers Tell About Architectures.

Nagios Plugin for dmesg Monitoring

So far I found no easy solution to monitor for Linux kernel messages. So here is a simple Nagios plugin to scan dmesg output for interesting stuff:

#!/bin/bash

SEVERITIES="err,alert,emerg,crit"
WHITELIST="microcode: |\
Firmware Bug|\
i8042: No controller|\
Odd, counter constraints enabled but no core perfctrs detected|\
Failed to access perfctr msr|\
echo 0 > /proc/sys"

# Check for critical dmesg lines from this day
date=$(date "+%a %b %e")
output=$(dmesg -T -l "$SEVERITIES" | egrep -v "$WHITELIST" | grep "$date" | tail -5)

if [ "$output" == "" ]; then
	echo "All is fine."
	exit 0
fi

echo "$output" | xargs
exit 1

"Features" of the script above:

  • It gives you the 5 most recent messages from today
  • It allows to whitelist common but useless errors in $WHITELIST
  • It uses "dmesg" to work when you already have disk I/O errors and to be faster than syslog parsing

This script helped a lot to early on detect I/O errors, recoverable as well as corruptions. It even worked when entire root partition wasn't readable anymore, because then the Nagios check failed with "NRPE: unable to read output" which indicated that dmesg didn't work anymore. By always showing all errors from the entire day one cannot miss recovered errors that happened in non-office hours.

Another good thing about the check is detecting OOM kills or fast spawning of processes.

Removing newlines with sed

My goal for today: I want to remember the official sed FAQ solution to replace multiple newlines:

sed ':a;N;$!ba;s/\n//g' file

to avoid spending a lot of time on it when I need it again.

Chef Gets Push in Q1/2014

Sysadvent features a puppetlabs sponsered article (yes, honestly, check the bottom of the page!) about chef enterprise getting push support. It is supposed to be included in the open source release in Q1/2014.

With this change you can use a push jobs cookbook to define jobs and an extended "knife" with new commands to start and query about jobs:

knife job start ...
knife job list

and

knife node status ...

will tell about job execution status on the remote node.

At a first glance it seems nice. Then again I feel worried when this is intended to get rid of SSH keys. Why do we need to get rid of them exactly? And in exchange for what?

Simple Chef to Nagios Hostgroup Export

When you are automatizing with chef and use plain Nagios for monitoring you will find duplication quite some configuration. One large part is the hostgroup definitions which usually map many of the chef roles. So if the roles are defined in chef anyway they should be sync'ed to Nagios.

Using "knife" one can extract the roles of a node like this

knife node show -a roles $node | grep -v "^roles:"

Scripting The Role Dumping

Note though that knife only shows roles that were applied on the server already. But this shouldn't be a big problem for a synchronization solution. Next step is to create a usable hostgroup definition in Nagios. To avoid colliding with existing hostgroups let's prefix the generated hostgroup names with "chef-". The only challenge is the regrouping of the role lists given per node by chef into host name lists per role. In Bash 4 using an fancy hash this could be done like this:

declare -A roles

for node in $(knife node list); do
   for role in $(knife node show -a roles $i |grep -v "roles" ); do
      roles["$role"]=${roles["$role"]}"$i "
   done
done

Given this it is easy to dump Icinga hostgroup definitions. For example

for role in ${!roles[*]}; do
   echo "define hostgroup {
   hostgroup_name chef-$role
   members ${roles[$role]}
}
"
done

That makes ~15 lines of shell script and a cronjob entry to integrate Chef with Nagios. Of course you also need to ensure that each host name provided by chef has a Nagios host definition. If you know how it resolves you could just dump a host definition while looping over the host list. In any case there is no excuse not to export the chef config :-)

Easy Migrating

Migrating to such an export is easy by using the "chef-" namespace prefix for generated hostgroups. This allows you to smoothly migrate existing Nagions definitions at your own pace. Be sure to only reload Nagios and not restart via cron and to do it at reasonable time to avoid breaking things.

What Job Offers Tell About Architectures #4

Part 4 of the Unix/Linux system administrator job offer and technologies survey. Here are several new companies. Note that some positions are actually developer positions. I chose those because they still told much about the software stack used. Check the "[src]" for the original reference, but know that it might be already offline.

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
Brose [src] AIX, Linux, Windows HA ABAP Oracle, MaxDB SAP - - - -
DATEV eG [src] - Java, Javascript DB2 J2EE, DOJO, JUnit, Mockito - - - -
DATEV eG [src] z/OS Cobol, Java, Javascript DB2, VSAM - - - - -
Elektrobit Automotive GmbH [src] Unix Java Oracle Tomcat, WebSphere, Spring, Eclipse RCP - - - -
Actano GmbH [src] Win, Mac OSX, Linux - MSSQL, Oracle, MySQL Firewall, Exchange - Nagios, Centreon vSphere, HyperV, AWS, Profitbricks -
Ratbacher [src] AIX, Linux - DB2, Oracle Tivoli Storage Manager IBM Power - - -
United Experts [src] Debian Lenny, Squeeze, Wheezy - MySQL, Postgresql Haproxy, Apache, Tomcat Network HW, Blades, Storages Nagios, Munin, Icinga, Zabbix Xen, KVM -
Texas Instruments [src] Solaris, Linux Shell, Python, Perl Oracle - x86 and SPARC HW components, Blades, SAN, NAS - - -
Media Saturn [src] Unix - - Tomcat, Citrix, Cluster, ITIL - - - -
easyCash [src] Linux, AIX Shell - Apache, Tomcat, JBoss, Weblogic, ITIL - Nagios - -

The Interesting Things

Nothing unexpected really. Well, except noone seems to use automation. All offers look very serious, as non-startup-like businesses should! Among them several AIX users and even a z/OS position offered by German tax services provider DATEV.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

What Job Offers Tell About Architectures #3

Part 3 of the Unix/Linux system administrator job offer and technologies survey. Here are 8 new companies:

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
Sunbeam GmbH [src] Debian, CentOS PHP MySQL Typo3, Piwik, Mantis, OpenX - - OpenVZ -
Air Berlin [src] Windows - Oracle 11g, MS SQL 2008 - - - - -
BearingPoint [src] Unix, Linux Shell, Java Oracle Apache, Tomcat - - - -
COC AG [src] Unix, Linux, Windows Server - ADS - - - VMWare, MS, Citrix -
dm Pharmacy [src] RHEL, Window Server 2008 Shell, Ruby, Java Varnish, Apache, JBoss, Tomcat - - - VMWare Chef or Puppet
Groupon GmbH [src] Linux - MySQL, Postgres Apache, Tomcat, Hudson, Jenkins, LDAP - - vSphere, VirtualBox cfengine2/3, Puppet
HUK-Coburg Insurance [src] - - DB2 DWH - - - -
InVision AG [src] Linux - Postgres, MySQL, Redis - - - - Chef, Puppet, Capistrano

The Interesting Things

Well none of the above companies needs monitoring :-)

And GroupOn is one of the few companies so far asking for cfengine. And for me the first time I heard about capistrano (requested by Invision AG). The github repository tells it is a:

"Remote multi-server automation tool http://www.capistranorb.com"

implemented in Ruby modelled from the Rake DSL. I'm worried that I missed this so long as it seems to be quite popular on github.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

Syndicate content Syndicate content