Blogs

What Job Offers Tell About Architectures #3

Part 3 of the Unix/Linux system administrator job offer and technologies survey. Here are 8 new companies:

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
Sunbeam GmbH [src] Debian, CentOS PHP MySQL Typo3, Piwik, Mantis, OpenX - - OpenVZ -
Air Berlin [src] Windows - Oracle 11g, MS SQL 2008 - - - - -
BearingPoint [src] Unix, Linux Shell, Java Oracle Apache, Tomcat - - - -
COC AG [src] Unix, Linux, Windows Server - ADS - - - VMWare, MS, Citrix -
dm Pharmacy [src] RHEL, Window Server 2008 Shell, Ruby, Java Varnish, Apache, JBoss, Tomcat - - - VMWare Chef or Puppet
Groupon GmbH [src] Linux - MySQL, Postgres Apache, Tomcat, Hudson, Jenkins, LDAP - - vSphere, VirtualBox cfengine2/3, Puppet
HUK-Coburg Insurance [src] - - DB2 DWH - - - -
InVision AG [src] Linux - Postgres, MySQL, Redis - - - - Chef, Puppet, Capistrano

The Interesting Things

Well none of the above companies needs monitoring :-)

And GroupOn is one of the few companies so far asking for cfengine. And for me the first time I heard about capistrano (requested by Invision AG). The github repository tells it is a:

"Remote multi-server automation tool http://www.capistranorb.com"

implemented in Ruby modelled from the Rake DSL. I'm worried that I missed this so long as it seems to be quite popular on github.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

PHP ini_set() Examples

Syntax of ini_set()

The ini_set() syntax is simple:

string ini_set ( string $varname , string $newvalue )

it is just a key value setter. The important question is which values can be set. Below you find a list of working examples. Please note that you cannot change all php.ini options especially those that need to be set before PHP initializes.

Useful Working ini_set() Examples

1. Enabling error display

On production sites you typically do not show errors in the page for usability and security reasons. But when you when you debug something live you might want to enable it temporarily and just for you:

# Assuming 5.44.33.22 is your IP...
if ( $_SERVER["REMOTE_ADDR"] == "5.44.33.22") {
    ini_set('display_errors', '1');
}

Note: you may want to combine this with

error_reporting(E_ALL | E_STRICT);

2. Changing Memory Limit

When you need to increase memory from within the code:

ini_set("memory_limit","1000M");

Note though that this might be prevent by a Suhosin hardended PHP installation.

3. Adding include paths

Normally this shouldn't be necessary. It is way cleaner to do it in php.ini, but if you bundle libraries and you administrator doesn't know:

<?php ini_set('include_path',ini_get('include_path').':../my-libs:');  ?>

When You Cannot Use ini_set()

For most php.ini settings you can't use ini_set(). To workaround consider deploying a .htaccess along with your code as this .htaccess can provide all PHP options to overwrite the default php.ini settings.

For example to change the HTTP POST limit add this line to a .htaccess read by your webserver:

php_value post_max_size 2500000

Note how the "php_value" prefix indicates settings for PHP. So simple syntax is

php_value <key name> <value>

What Job Offers Tell About Architectures #2

It is time for another round of having a look which technologies companies announce with job openings for system administrators.

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
1&1 Shared Hosting [src] Debian Perl, Bash - Apache, DRBD, Debian Packaging - - - -
1&1 Virtualisation [src] Linux Perl, Python - - NAS, SAN, iSCSI, NFS - VMWare, Xen, KVM, OpenStack, OpenNebula -
1&1 Monitoring [src] Linux, Windows Server Java, C/C++, Perl, Bash, Ruby - - - - - -
1&1 [src] Linux Perl, Bash, Python MySQL, Oracle, NoSQL Apache, Tomcat, Java - - - -
1&1 [src] Linux Java - Apache, Tomcat, JBoss - - - -
1&1 gmx.de web.de [src] Linux Perl, Bash, Python MySQL, Oracle, NoSQL, Cassandra - - - - Puppet
buch.de [src] - - MySQL, MS SQL DRBD, heartbeat - - - -
First Colo [src] Linux Bash, PHP, Perl, Python Postgres, MySQL - Cisco, Juniper - - -
IKB Bank [src] SLES 10/11/12, Solaris 8/10 Bash, Perl Postgres, MySQL - Fibre Channel SAN, Storages: EMC, HP, IBM - - -
s'Oliver [src] AIX - - SAP, Tivoli TSM IBM SAN - PowerVM -

The Interesting Things

Well as you can see a lot of 1&1 offers. With 1&1 being on of the large successful German ISPs running the leading mail portals GMX and Web.de it is interesting to see their technology requirements. They seem to be focussing on Java, MySQL and Oracle with NoSQL and Apache Cassandra being the youngest startup-ish tool used. So for me 1&1 makes a somewhat old-school impression, which of course is not necessarily bad. As they indicate no standard monitoring solution I guess they use a proprietory or commercial solution.

What I found noteworthy about the other companies is the usage of DRBD. It's more commonly used than I'd expect so far.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

Large Website Technology Changes in October 2013

Last month I did a first indexing of mostly request header, HTML and DNS based information about the top 200 sites listed by Alexa and the top 100 German websites. All the information is freely available and extracted from the website responses.

The detailed results can be found here:

What Changed In A Month?

Now roughly a month later I repeated the scan to see what changes do happen and what trends might be noticable:

DNS-Prefetching THE HTML header based DNS prefetching is spreading. Two of the large adult content sites added it and Chinese search engine baidu.com introduced it too. As in my experience it provides a small but measurable latency improvement on many sites I'd guess usage will further spread...

Added by baidu.com, pornhub.com, tube8.com, www.wer-kennt-wen.de
Removed by taobao.com

IPv6 IPv6 support did not spread in the last month. Just two sides that dropped their AAAA records.

Removed by www.mail.ru, www.volkswagen.de

Varnish One of the most significant changes is more and more sites announcing the usage of Varnish as a cache. This is sometimes combined with a CDN, sometimes without.

Introduced by bbc.co.uk, www.otto.de, www.pinterest.com, www.wikipedia.org

Hiding Server Version Two more websites are now hiding their Apache version.

Removed version now: www.weibo.com, www.ftd.de

XSS Header XSS headers are not widely spread. No real change in adoption.

Added by www.craigslist.org
Removed by www.pinterest.com

Caution!

All the results listed above are based on a simple scanning script. The results present a snapshot of the websites and a single response only. This is of course not necessarily an indicating for what techniques the site uses in daily operations!

Learning from Job Offers

Every other day you get one, fly over it and delete it. In a way each time it says exactly the same. Or does it?

Surely the different companies you read job offers from are using different technologies. Actually when announcing the position they are at their weakest time. They have to admit which technologies they use, how heterogenous they are and sometimes how old-style they are.

I think I'll from time to time compile some positions available online (not offers I got!) of mostly German and maybe more Berlin located companies and add them to an ever growing list "What Job Offers Tell". Below you find the first 10 company offers with their data.

Todays List

Company Link OS Languages Databases Software Hardware Monitoring Hosting Automation
Fraunhofer HHI [src] Ubuntu, Redhat, Debian C# - Windows AD, Windows Terminal Server, DFS Fibre Channel - vSphere 5 -
Idealo.de [src] Debian, Redhat - - - - Icinga, Cacti, NewRelic - -
Idealo.de [src] Debian - NoSQL, MongoDB LAMP - Nagios, Zookeeper, Corosync KVM -
KPMG [src] Windows Server 2003/2008 - MS SQL 2005/2008, SQL BI MS IIS - - - -
Lusini.de [src] Linux Node.js MongoDB nginx, Varnish, Elastic Search - NewRelic - Puppet, SaltStack
maxdome.de [src] Redhat PHP MySQL, Postgres, Redis, CouchDB Tomcat, JBoss, Apache, nginx - - Cloud Puppet, Foreman, Chef, Rex
Springer Online [src] CentOS, Redhat Bash, Python, Ruby MySQL, Postgres, MongoDB Apache, HaProxy, JBoss, Tomcat, Nginx, Varnish F5, Cisco Icinga, Graphite KVM, VMWare, Xen, AWS chef, Puppet
Teufel [src] Windows Powershell, Bash Exchange - - - VMWare, Hyper V chef, Puppet
T Systems [src] Linux Shell, Perl, PHP MySQL, Oracle heartbeat, DRBD, Apache, Tomcat, JBoss, Weblogic - - - -
Zalando [src] Windows - - MS ADS SAN - - -

The Interesting Parts

It gets interesting where positions go into details like with T-Systems using heartbeat and DRBD or Springer mentioning the real-time graphing engine Graphite. I also like maxdome using four different automation tools: Puppet, Foreman, Chef and Rex. That's two to many. A more exotic thing is Idealo.de using Apache Zookeeper, or is it more common than I think?

On the other hand some other candidates either have no automation needs, or the lack of hints indicates self-made automation or none at all.

Stay tuned for the next set of companies!

Feedly Support for Liferea

A few days ago Feedly sent sandbox accounts around to developers that registered to be interested in the API. I intend to try to implement it and if everything works well to request a production API token. Stay tuned!

Meanwhile I'm working on adding Tiny Tiny RSS API v1.7.6 based features: subscribing and unsubscribing feeds from within Liferea.

New Unstable Branch for Liferea 1.11: Support for InoReader and Reedah

Yesterday I started a new 1.11 release branch in git. And my first goal is to introduce working support for the two free online aggregators

Both have a Google Reader like JSON API, so code reusage is possible. The challenge at the moment is to have as much shared code as possible while still covering the differences.

And each time your hear someone claim this or that online service is Google Reader API compatible ask: which the Google Reader Atom API or the JSON API.

Actually every combination: pure JSON, mixed Atom/JSON and Atom APIs are out there. And noone is encoding a simple thing like the read state of an item the same way. What usually is the same like back with Google Reader are the names of the API endpoints :-)

Still normalizing all the APIs is a challenge itself, which is why I'm working on it. If at the end Liferea users will have a choice I'll be happy. If you are using either of those two InoReader or Reedah please leave a short note in the comments! Both readers were requested in comments to earlier post and it helps decisions a lot to know what everyone is using!

HowTo: Munin and rrdcached on Ubuntu 12.04

Let's expect you already have Munin installed and working and you want to reduce disk I/O and improve responsiveness by adding rrdcached... Here are the complete steps to integrate rrdcached:

Basic Installation

First install the stock package

apt-get install rrdcached

and integrate it with Munin:

  1. Enable the rrdcached socket line in /etc/munin/munin.conf
  2. Disable munin-html and munin-graph calls in /usr/bin/munin-cron
  3. Create /usr/bin/munin-graph with
    #!/bin/bash
    
    nice /usr/share/munin/munin-html $@ || exit 1
    
    nice /usr/share/munin/munin-graph --cron $@ || exit 1 

    and make it executable

  4. Add a cron job (e.g. to /etc/cron.d/munin) to start munin-graph:
    10 * * * *      munin if [ -x /usr/bin/munin-graph ]; then /usr/bin/munin-graph; fi

The Critical Stuff

To get Munin to use rrdcached on Ubuntu 12.04 ensure to follow these vital steps:

  1. Add "-s <webserver group>" to $OPT in /etc/init.d/rrdcached (in front of the first -l switch)
  2. Change "-b /var/lib/rrdcached/db/" to "-b /var/lib/munin" (or wherever you keep your RRDs)


So a patched default Debian/Ubuntu with Apache /etc/init.d/rrdcached would have

OPTS="-s www-data -l unix:/var/run/rrdcached.sock"
OPTS="$OPTS -j /var/lib/rrdcached/journal/ -F"
OPTS="$OPTS -b /var/lib/munin/ -B"

If you do not set the socket user with "-s" you will see "Permission denied" in /var/log/munin/munin-cgi-graph.log

[RRD ERROR] Unable to graph /var/lib/munin/
cgi-tmp/munin-cgi-graph/[...].png : Unable to connect to rrdcached: 
Permission denied

If you do not change the rrdcached working directory you will see "rrdc_flush" errors in your /var/log/munin/munin-cgi-graph.log

[RRD ERROR] Unable to graph /var/lib/munin/
cgi-tmp/munin-cgi-graph/[...].png : 
rrdc_flush (/var/lib/munin/[...].rrd) failed with status -1.

Some details on this can be found in the Munin wiki.

Liferea Code Repo Moved to github

I moved the source repo away from SourceForge away to GitHub.
It is currently located here:

https://github.com/lwindolf/liferea

If in doubt always follow the "Code" link from the website to find the repo.

Sorry, if this causes troubles for you. I'll contact all with current git write
access directly to see how we can continue on github and who will be able
to merge.

Please keep contributing! I think with github this can actually become
easier and more developers are familiar with its best practices.

GLib GRegex Regular Expression Cheat Sheet

Glib supports PCRE based regular expressions since v2.14 with the GRegex class.

Usage

GError *err = NULL;
GMatchInfo *matchInfo;
GRegex *regex;
   
regex = g_regex_new ("text", 0, 0, &err);
// check for compilation errors here!
     
g_regex_match (regex, "Some text to match", 0, &matchInfo);

Not how g_regex_new() gets the pattern as first parameter without any regex delimiters. As the regex is created separately it can and should be reused.

Checking if a GRegex did match

Above example just ran the regular expression, but did not test for matching. To simply test for a match add something like this:

if (g_match_info_matches (matchInfo))
    g_print ("Text found!\n");

Extracting Data

If you are interested in data matched you need to use matching groups and need to iterate over the matches in the GMatchInfo structure. Here is an example (without any error checking):

regex = g_regex_new (" mykey=(\w+) ", 0, 0, &err);   
g_regex_match (regex, content, 0, &matchInfo);

while (g_match_info_matches (matchInfo)) {
   gchar *result = g_match_info_fetch (matchInfo, 0);

   g_print ("mykey=%s\n", result);
         
   g_match_info_next (matchInfo, &err);
   g_free (result);
}

Easy String Splitting

Another nice feature in Glib is regex based string splitting with g_regex_split() or g_regex_split_simple():

gchar **results = g_regex_split_simple ("\s+", 
       "White space separated list", 0, 0);

Use g_regex_split for a precompiled regex or use the "simple" function to just pass the pattern.

Syndicate content