Blogs

Removing newlines with sed

My goal for today: I want to remember the official sed FAQ solution to replace multiple newlines:

sed ':a;N;$!ba;s/\n//g' file

to avoid spending a lot of time on it when I need it again.

Chef Gets Push in Q1/2014

Sysadvent features a puppetlabs sponsered article (yes, honestly, check the bottom of the page!) about chef enterprise getting push support. It is supposed to be included in the open source release in Q1/2014.

With this change you can use a push jobs cookbook to define jobs and an extended "knife" with new commands to start and query about jobs:

knife job start ...
knife job list

and

knife node status ...

will tell about job execution status on the remote node.

At a first glance it seems nice. Then again I feel worried when this is intened to get rid of SSH keys. Why do we need to get rid of them exactly? And in exchange for what?

Simple Chef to Nagios Hostgroup Export

When you are automatizing with chef and use plain Nagios for monitoring you will find duplication quite some configuration. One large part is the hostgroup definitions which usually map many of the chef roles. So if the roles are defined in chef anyway they should be sync'ed to Nagios.

Using "knife" one can extract the roles of a node like this

knife node show -a roles $node | grep -v "^roles:"

Scripting The Role Dumping

Note though that knife only shows roles that were applied on the server already. But this shouldn't be a big problem for a synchronization solution. Next step is to create a usable hostgroup definition in Nagios. To avoid colliding with existing hostgroups let's prefix the generated hostgroup names with "chef-". The only challenge is the regrouping of the role lists given per node by chef into host name lists per role. In Bash 4 using an fancy hash this could be done like this:

declare -A roles

for node in $(knife node list); do
   for role in $(knife node show -a roles $i |grep -v "roles" ); do
      roles["$role"]=${roles["$role"]}"$i "
   done
done

Given this it is easy to dump Icinga hostgroup definitions. For example

for role in ${!roles[*]}; do
   echo "define hostgroup {
   hostgroup_name chef-$role
   members ${roles[$role]}
}
"
done

That makes ~15 lines of shell script and a cronjob entry to integrate Chef with Nagios. Of course you also need to ensure that each host name provided by chef has a Nagios host definition. If you know how it resolves you could just dump a host definition while looping over the host list. In any case there is no excuse not to export the chef config :-)

Easy Migrating

Migrating to such an export is easy by using the "chef-" namespace prefix for generated hostgroups. This allows you to smoothly migrate existing Nagions definitions at your own pace. Be sure to only reload Nagios and not restart via cron and to do it at reasonable time to avoid breaking things.

What Job Offers Tell About Architectures #4

Part 4 of the Unix/Linux system administrator job offer and technologies survey. Here are several new companies. Note that some positions are actually developer positions. I chose those because they still told much about the software stack used. Check the "[src]" for the original reference, but know that it might be already offline.

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
Brose [src] AIX, Linux, Windows HA ABAP Oracle, MaxDB SAP - - - -
DATEV eG [src] - Java, Javascript DB2 J2EE, DOJO, JUnit, Mockito - - - -
DATEV eG [src] z/OS Cobol, Java, Javascript DB2, VSAM - - - - -
Elektrobit Automotive GmbH [src] Unix Java Oracle Tomcat, WebSphere, Spring, Eclipse RCP - - - -
Actano GmbH [src] Win, Mac OSX, Linux - MSSQL, Oracle, MySQL Firewall, Exchange - Nagios, Centreon vSphere, HyperV, AWS, Profitbricks -
Ratbacher [src] AIX, Linux - DB2, Oracle Tivoli Storage Manager IBM Power - - -
United Experts [src] Debian Lenny, Squeeze, Wheezy - MySQL, Postgresql Haproxy, Apache, Tomcat Network HW, Blades, Storages Nagios, Munin, Icinga, Zabbix Xen, KVM -
Texas Instruments [src] Solaris, Linux Shell, Python, Perl Oracle - x86 and SPARC HW components, Blades, SAN, NAS - - -
Media Saturn [src] Unix - - Tomcat, Citrix, Cluster, ITIL - - - -
easyCash [src] Linux, AIX Shell - Apache, Tomcat, JBoss, Weblogic, ITIL - Nagios - -

The Interesting Things

Nothing unexpected really. Well, except noone seems to use automation. All offers look very serious, as non-startup-like businesses should! Among them several AIX users and even a z/OS position offered by German tax services provider DATEV.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

What Job Offers Tell About Architectures #3

Part 3 of the Unix/Linux system administrator job offer and technologies survey. Here are 8 new companies:

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
Sunbeam GmbH [src] Debian, CentOS PHP MySQL Typo3, Piwik, Mantis, OpenX - - OpenVZ -
Air Berlin [src] Windows - Oracle 11g, MS SQL 2008 - - - - -
BearingPoint [src] Unix, Linux Shell, Java Oracle Apache, Tomcat - - - -
COC AG [src] Unix, Linux, Windows Server - ADS - - - VMWare, MS, Citrix -
dm Pharmacy [src] RHEL, Window Server 2008 Shell, Ruby, Java Varnish, Apache, JBoss, Tomcat - - - VMWare Chef or Puppet
Groupon GmbH [src] Linux - MySQL, Postgres Apache, Tomcat, Hudson, Jenkins, LDAP - - vSphere, VirtualBox cfengine2/3, Puppet
HUK-Coburg Insurance [src] - - DB2 DWH - - - -
InVision AG [src] Linux - Postgres, MySQL, Redis - - - - Chef, Puppet, Capistrano

The Interesting Things

Well none of the above companies needs monitoring :-)

And GroupOn is one of the few companies so far asking for cfengine. And for me the first time I heard about capistrano (requested by Invision AG). The github repository tells it is a:

"Remote multi-server automation tool http://www.capistranorb.com"

implemented in Ruby modelled from the Rake DSL. I'm worried that I missed this so long as it seems to be quite popular on github.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

PHP ini_set() Examples

Syntax of ini_set()

The ini_set() syntax is simple:

string ini_set ( string $varname , string $newvalue )

it is just a key value setter. The important question is which values can be set. Below you find a list of working examples. Please note that you cannot change all php.ini options especially those that need to be set before PHP initializes.

Useful Working ini_set() Examples

1. Enabling error display

On production sites you typically do not show errors in the page for usability and security reasons. But when you when you debug something live you might want to enable it temporarily and just for you:

# Assuming 5.44.33.22 is your IP...
if ( $_SERVER["REMOTE_ADDR"] == "5.44.33.22") {
    ini_set('display_errors', '1');
}

Note: you may want to combine this with

error_reporting(E_ALL | E_STRICT);

2. Changing Memory Limit

When you need to increase memory from within the code:

ini_set("memory_limit","1000M");

Note though that this might be prevent by a Suhosin hardended PHP installation.

3. Adding include paths

Normally this shouldn't be necessary. It is way cleaner to do it in php.ini, but if you bundle libraries and you administrator doesn't know:

<?php ini_set('include_path',ini_get('include_path').':../my-libs:');  ?>

When You Cannot Use ini_set()

For most php.ini settings you can't use ini_set(). To workaround consider deploying a .htaccess along with your code as this .htaccess can provide all PHP options to overwrite the default php.ini settings.

For example to change the HTTP POST limit add this line to a .htaccess read by your webserver:

php_value post_max_size 2500000

Note how the "php_value" prefix indicates settings for PHP. So simple syntax is

php_value <key name> <value>

What Job Offers Tell About Architectures #2

It is time for another round of having a look which technologies companies announce with job openings for system administrators.

Todays List

Company Reference OS Languages Databases Software Hardware Monitoring Hosting Automation
1&1 Shared Hosting [src] Debian Perl, Bash - Apache, DRBD, Debian Packaging - - - -
1&1 Virtualisation [src] Linux Perl, Python - - NAS, SAN, iSCSI, NFS - VMWare, Xen, KVM, OpenStack, OpenNebula -
1&1 Monitoring [src] Linux, Windows Server Java, C/C++, Perl, Bash, Ruby - - - - - -
1&1 [src] Linux Perl, Bash, Python MySQL, Oracle, NoSQL Apache, Tomcat, Java - - - -
1&1 [src] Linux Java - Apache, Tomcat, JBoss - - - -
1&1 gmx.de web.de [src] Linux Perl, Bash, Python MySQL, Oracle, NoSQL, Cassandra - - - - Puppet
buch.de [src] - - MySQL, MS SQL DRBD, heartbeat - - - -
First Colo [src] Linux Bash, PHP, Perl, Python Postgres, MySQL - Cisco, Juniper - - -
IKB Bank [src] SLES 10/11/12, Solaris 8/10 Bash, Perl Postgres, MySQL - Fibre Channel SAN, Storages: EMC, HP, IBM - - -
s'Oliver [src] AIX - - SAP, Tivoli TSM IBM SAN - PowerVM -

The Interesting Things

Well as you can see a lot of 1&1 offers. With 1&1 being on of the large successful German ISPs running the leading mail portals GMX and Web.de it is interesting to see their technology requirements. They seem to be focussing on Java, MySQL and Oracle with NoSQL and Apache Cassandra being the youngest startup-ish tool used. So for me 1&1 makes a somewhat old-school impression, which of course is not necessarily bad. As they indicate no standard monitoring solution I guess they use a proprietory or commercial solution.

What I found noteworthy about the other companies is the usage of DRBD. It's more commonly used than I'd expect so far.

Note: You can find the complete index of all companies so far here: What Job Offers Tell About Architectures.

Large Website Technology Changes in October 2013

Last month I did a first indexing of mostly request header, HTML and DNS based information about the top 200 sites listed by Alexa and the top 100 German websites. All the information is freely available and extracted from the website responses.

The detailed results can be found here:

What Changed In A Month?

Now roughly a month later I repeated the scan to see what changes do happen and what trends might be noticable:

DNS-Prefetching THE HTML header based DNS prefetching is spreading. Two of the large adult content sites added it and Chinese search engine baidu.com introduced it too. As in my experience it provides a small but measurable latency improvement on many sites I'd guess usage will further spread...

Added by baidu.com, pornhub.com, tube8.com, www.wer-kennt-wen.de
Removed by taobao.com

IPv6 IPv6 support did not spread in the last month. Just two sides that dropped their AAAA records.

Removed by www.mail.ru, www.volkswagen.de

Varnish One of the most significant changes is more and more sites announcing the usage of Varnish as a cache. This is sometimes combined with a CDN, sometimes without.

Introduced by bbc.co.uk, www.otto.de, www.pinterest.com, www.wikipedia.org

Hiding Server Version Two more websites are now hiding their Apache version.

Removed version now: www.weibo.com, www.ftd.de

XSS Header XSS headers are not widely spread. No real change in adoption.

Added by www.craigslist.org
Removed by www.pinterest.com

Caution!

All the results listed above are based on a simple scanning script. The results present a snapshot of the websites and a single response only. This is of course not necessarily an indicating for what techniques the site uses in daily operations!

Learning from Job Offers

Every other day you get one, fly over it and delete it. In a way each time it says exactly the same. Or does it?

Surely the different companies you read job offers from are using different technologies. Actually when announcing the position they are at their weakest time. They have to admit which technologies they use, how heterogenous they are and sometimes how old-style they are.

I think I'll from time to time compile some positions available online (not offers I got!) of mostly German and maybe more Berlin located companies and add them to an ever growing list "What Job Offers Tell". Below you find the first 10 company offers with their data.

Todays List

Company Link OS Languages Databases Software Hardware Monitoring Hosting Automation
Fraunhofer HHI [src] Ubuntu, Redhat, Debian C# - Windows AD, Windows Terminal Server, DFS Fibre Channel - vSphere 5 -
Idealo.de [src] Debian, Redhat - - - - Icinga, Cacti, NewRelic - -
Idealo.de [src] Debian - NoSQL, MongoDB LAMP - Nagios, Zookeeper, Corosync KVM -
KPMG [src] Windows Server 2003/2008 - MS SQL 2005/2008, SQL BI MS IIS - - - -
Lusini.de [src] Linux Node.js MongoDB nginx, Varnish, Elastic Search - NewRelic - Puppet, SaltStack
maxdome.de [src] Redhat PHP MySQL, Postgres, Redis, CouchDB Tomcat, JBoss, Apache, nginx - - Cloud Puppet, Foreman, Chef, Rex
Springer Online [src] CentOS, Redhat Bash, Python, Ruby MySQL, Postgres, MongoDB Apache, HaProxy, JBoss, Tomcat, Nginx, Varnish F5, Cisco Icinga, Graphite KVM, VMWare, Xen, AWS chef, Puppet
Teufel [src] Windows Powershell, Bash Exchange - - - VMWare, Hyper V chef, Puppet
T Systems [src] Linux Shell, Perl, PHP MySQL, Oracle heartbeat, DRBD, Apache, Tomcat, JBoss, Weblogic - - - -
Zalando [src] Windows - - MS ADS SAN - - -

The Interesting Parts

It gets interesting where positions go into details like with T-Systems using heartbeat and DRBD or Springer mentioning the real-time graphing engine Graphite. I also like maxdome using four different automation tools: Puppet, Foreman, Chef and Rex. That's two to many. A more exotic thing is Idealo.de using Apache Zookeeper, or is it more common than I think?

On the other hand some other candidates either have no automation needs, or the lack of hints indicates self-made automation or none at all.

Stay tuned for the next set of companies!

Feedly Support for Liferea

A few days ago Feedly sent sandbox accounts around to developers that registered to be interested in the API. I intend to try to implement it and if everything works well to request a production API token. Stay tuned!

Meanwhile I'm working on adding Tiny Tiny RSS API v1.7.6 based features: subscribing and unsubscribing feeds from within Liferea.

Syndicate content