Sysadmin

Puppet Agent Noop Pitfalls

The puppet agent command has a --noop switch that allows you to perform a dry-run of your Puppet code.

puppet agent -t --noop

It doesn't change anything, it just tells you what it would change. More or less exact due to the nature of dependencies that might come into existance by runtime changes. But it is pretty helpful and all Puppet users I know use it from time to time.

Unexpected Things

But there are some unexpected things about the noop mode:

  1. A --noop run does trigger the report server.
  2. The --noop run rewrites the YAML state files in /var/lib/puppet
  3. And there is no state on the local machine that gives you the last "real" run result after you overwrite the state files with the --noop run.

Why might this be a problem?

Or the other way around: why Puppet think this is not a problem? Probably because Puppet as an automation tool should overwrite and the past state doesn't really matter. If you use PE or Puppet with PuppetDB or Foreman you have an reporting for past runs anyway, so no need to have a history on the Puppet client.

Why I still do not like it: it avoids having safe and simple local Nagios checks. Using the state YAML you might want to build a simple script checking for run errors. Because you might want a Nagios alert about all errors that appear. Or about hosts that did not run Puppet for quite some time (for example I wanted to disable Puppet on a server for some action and forgot to reenable). Such a check reports false positives each time someone does a --noop run until the next normal run. This hides errors.

Of course you can build all this with cool Devops style SQL/REST/... queries to PuppetDB/Foreman, but checking state locally seems a bit more the old-style robust and simpler sysadmin way. Actively asking the Puppet master or report server for the client state seems wrong. The client should know too.

From a software usability perspective I do not expect a tool do change it's state when I pass --noop. It's unexpected. Of course the documentation is carefull phrased:

Use 'noop' mode where the daemon runs in a no-op or dry-run mode. This is useful for seeing what changes Puppet will make without actually executing the changes.

Puppet Apply Only Specific Classes

If you want to apply Puppet changes in an selective manner you can run

puppet apply -t --tags Some::Class

on the client node to only run the single class named "Some::Class".

Why does this work? Because Puppet automatically creates tags for all classes you have. Ensure to upper-case all parts of the class name, because even if you actual Ruby class is "some::class" the Puppet tag will be "Some::Class".

Redis Performance Debugging

Here are some simple hints on debugging Redis performance issues.

Monitoring Live Redis Queries

Run the "monitor" command to see queries as they are sent against an Redis instance. Do not use on high traffic instance!

redis-cli monitor

The output looks like this

redis 127.0.0.1:6379> MONITOR
OK
1371241093.375324 "monitor"
1371241109.735725 "keys" "*"
1371241152.344504 "set" "testkey" "1"
1371241165.169184 "get" "testkey"

Analyzing Slow Commands

When there are too many queries better use "slowlog" to see the top slow queries running against your Redis instance:

slowlog get 25		# print top 25 slow queries
slowlog len		
slowlog reset

Debugging Latency

If you suspect latency to be an issue use "redis-cli" built-in support for latency measuring. First measure system latency on your Redis server with

redis-cli --intrinsic-latency 100

and then sample from your Redis clients with

redis-cli --latency -h <host> -p <port>

If you have problems with high latency check if transparent huge pages are disabled. Disable it with

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Check Background Save Settings

If your instance seemingly freezes peridiocally you probably have background dumping enabled.

grep ^save /etc/redis/redis.conf

Comment out all save lines and setup a cron job to do dumping or a Redis slave who can dump whenever he wants to.

Alternatively you can try to mitigate the effect using the "no-appendfsync-on-rewrite" option (set to "yes") in redis.conf.

Check fsync Setting

Per default Redis runs fsync() every 1s. Other possibilities are "always" and "no".

grep ^appendfsync /etc/redis/redis.conf

So if you do not care about DB corruption you might want to set "no" here.

Puppet Check ERBs for Dynamic Scoping

If you ever need to upgrade a code base to Puppet 3.0 and strip all dynamic scoping from your templates:

for file in $( find . -name "*.erb" | sort); do 
    echo "------------ [ $file ]"; 
    if grep -q "<%[^>]*$" $file; then 
        content=$(sed '/<%/,/%>/!d' $file); 
    else
        content=$(grep "<%" $file); 
    fi;
    echo "$content" | egrep "(.each|if |%=)" | egrep -v "scope.lookupvar|@|scope\["; 
done

This is of course just a fuzzy match, but should catch quite some of the dynamic scope expressions there are. The limits of this solution are:

  • false positives on loop and declared variables that must not be scoped
  • and false negatives when mixing of correct scope and missing scope in the same line.

So use with care.

HowTo: Implement Consistent Hashing with Different memcached Bindings

A short HowTo on memcached consistent hashing. Of course also works with memcached protocol compatible software as CouchBase, MySQL...

Papers

Papers to read to learn about what consistent hashing is about:

Consistent Hashing with nginx

 upstream somestream {
      consistent_hash $request_uri;
      server 10.0.0.1:11211;
      server 10.0.0.2:11211;
      ...
    }

Consistent Hashing with PHP

Note: the order of setOption() and addServers() is important. When using OPT_LIBKETAMA_COMPATIBLE
the hashing is compatible with all other runtimes using libmemcached.

$memcached = new Memcached();
$memcached->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
$memcached->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$memcached->addServers($servers);

Consistent Hashing in Perl

As in PHP the order of setOptions() and addServers() matters. After all both languages use the same library in the background, so behaviour is the same.

$m = new Memcached('mymemcache');
$m->setOptions(array(
   ...
   Memcached::OPT_LIBKETAMA_COMPATIBLE => true,
   Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_CONSISTENT,
   ...
));
$m->addServers(...);

Puppet: List Changed Files

If you want to know which files where changed by puppet in the last days:

cd /var/lib/puppet
for i in $(find clientbucket/ -name paths); do
	echo "$(stat -c %y $i | sed 's/\..*//')       $(cat $i)";
done | sort -n

will give you an output like

[...]
2015-02-10 12:36:25       /etc/resolv.conf
2015-02-17 10:52:09       /etc/bash.bashrc
2015-02-20 14:48:18       /etc/snmp/snmpd.conf
2015-02-20 14:50:53       /etc/snmp/snmpd.conf
[...]

Debugging dovecot ACL Shared Mailboxes Not Showing in Thunderbird

When you can't get ACL shared mailboxes visible with Dovecot and Thunderbird here are some debugging tipps:

  1. Thunderbird fetches the ACLs on startup (and maybe at some other interval). So for testing restart Thunderbird on each change you make.
  2. Ensure the shared mailboxes index can be written. You probably have it configured like
    plugin {
      acl_shared_dict = file:/var/lib/dovecot/db/shared-mailboxes.db
    }

    Check if such a file was created and is populated with new entries when you add ACLs from the mail client. As long as entries do not appear here, nothing can work.

  3. Enable debugging in the dovecot log or use the "debug" flag and check the ACLs for the user who should see a shared mailbox like this:
    doveadm acl debug -u [email protected] shared/users/box
    • Watch out for missing directories
    • Watch out for permission issues
    • Watch out for strangely created paths this could hint a misconfigured namespace prefix

Getting rid of Bash Ctrl+R

Today was a good day, as I stumbled over this post (at http://architects.dzone.com) hinting on the following bash key bindings:

bind '"\e[A":history-search-backward'
bind '"\e[B":history-search-forward'

It changes the behaviour of the up and down cursor keys to not go blindly through the history but only through items matching the current prompt. Of course at the disadvantage of having to clear the line to go through the full history. But as this can be achieved by a Ctrl-C at any time it is still preferrable to Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R Ctrl+R ....

How Common Are HTTP Security Headers Really?

A recent issue of the German iX magazin featured an article on improving end user security by enabling HTTP security headers

  • X-XSS-Protection,
  • X-Content-Type-Options MIME type sniffing,
  • Content-Security-Policy,
  • X-Frame-Options,
  • and HSTS Strict-Transport-Security.

The article gave the impression of all of them quite common and a good DevOps being unreasonable not implementing them immediately if the application supports them without problems.

This lead me to check my monthly domain scan results of April 2014 on who is actually using which header on their main pages. Results as always limited to top 200 Alexa sites and all larger German websites.

Usage of X-XSS-Protection

Header visible for only 14 of 245 (5%) of the scanned websites. As 2 are just disabling the setting it is only 4% of the websites enabling it.

Website Header
www.adcash.com X-XSS-Protection: 1; mode=block
www.badoo.com X-XSS-Protection: 1; mode=block
www.blogger.com X-XSS-Protection: 1; mode=block
www.blogspot.com X-XSS-Protection: 1; mode=block
www.facebook.com X-XSS-Protection: 0
www.feedburner.com X-XSS-Protection: 1; mode=block
www.github.com X-XSS-Protection: 1; mode=block
www.google.de X-XSS-Protection: 1; mode=block
www.live.com X-XSS-Protection: 0
www.meinestadt.de X-XSS-Protection: 1; mode=block
www.openstreetmap.org X-XSS-Protection: 1; mode=block
www.tape.tv X-XSS-Protection: 1; mode=block
www.xing.de X-XSS-Protection: 1; mode=block; report=https://www.xing.com/tools/xss_reporter
www.youtube.de X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube

Usage of X-Content-Type-Options

Here 15 of 245 websites (6%) enable the option.

Website Header
www.blogger.com X-Content-Type-Options: nosniff
www.blogspot.com X-Content-Type-Options: nosniff
www.deutschepost.de X-Content-Type-Options: NOSNIFF
www.facebook.com X-Content-Type-Options: nosniff
www.feedburner.com X-Content-Type-Options: nosniff
www.github.com X-Content-Type-Options: nosniff
www.linkedin.com X-Content-Type-Options: nosniff
www.live.com X-Content-Type-Options: nosniff
www.meinestadt.de X-Content-Type-Options: nosniff
www.openstreetmap.org X-Content-Type-Options: nosniff
www.spotify.com X-Content-Type-Options: nosniff
www.tape.tv X-Content-Type-Options: nosniff
www.wikihow.com X-Content-Type-Options: nosniff
www.wikipedia.org X-Content-Type-Options: nosniff
www.youtube.de X-Content-Type-Options: nosniff

Usage of Content-Security-Policy

Actually only 1 website in the top 200 Alexa ranked websites uses CSP and this lonely site is github. The problem with CSP obviously being the necessity to have a clear structure for the origin domains of the site elements. And the less advertisments and tracking pixels you have the easier it becomes...

Website Header
www.github.com Content-Security-Policy: default-src *; script-src https://github.global.ssl.fastly.net https://ssl.google-analytics.com https://collector-cdn.github.com; style-src 'self' 'unsafe-inline' 'unsafe-eval' https://github.global.ssl.fastly.net; object-src https://github.global.ssl.fastly.net

Usage of X-Frame-Options

The X-Frame-Options header is currently delivered by 43 of 245 websites (17%).

Website Header
www.adcash.com X-Frame-Options: SAMEORIGIN
www.adf.ly X-Frame-Options: SAMEORIGIN
www.avg.com X-Frame-Options: SAMEORIGIN
www.badoo.com X-Frame-Options: DENY
www.battle.net X-Frame-Options: SAMEORIGIN
www.blogger.com X-Frame-Options: SAMEORIGIN
www.blogspot.com X-Frame-Options: SAMEORIGIN
www.dailymotion.com X-Frame-Options: deny
www.deutschepost.de X-Frame-Options: SAMEORIGIN
www.ebay.de X-Frame-Options: SAMEORIGIN
www.facebook.com X-Frame-Options: DENY
www.feedburner.com X-Frame-Options: SAMEORIGIN
www.github.com X-Frame-Options: deny
www.gmx.de X-Frame-Options: deny
www.gmx.net X-Frame-Options: deny
www.google.de X-Frame-Options: SAMEORIGIN
www.groupon.de X-Frame-Options: SAMEORIGIN
www.imdb.com X-Frame-Options: SAMEORIGIN
www.indeed.com X-Frame-Options: SAMEORIGIN
www.instagram.com X-Frame-Options: SAMEORIGIN
www.java.com X-Frame-Options: SAMEORIGIN
www.linkedin.com X-Frame-Options: SAMEORIGIN
www.live.com X-Frame-Options: deny
www.mail.ru X-Frame-Options: SAMEORIGIN
www.mozilla.org X-Frame-Options: DENY
www.netflix.com X-Frame-Options: SAMEORIGIN
www.openstreetmap.org X-Frame-Options: SAMEORIGIN
www.oracle.com X-Frame-Options: SAMEORIGIN
www.paypal.com X-Frame-Options: SAMEORIGIN
www.pingdom.com X-Frame-Options: SAMEORIGIN
www.skype.com X-Frame-Options: SAMEORIGIN
www.skype.de X-Frame-Options: SAMEORIGIN
www.softpedia.com X-Frame-Options: SAMEORIGIN
www.soundcloud.com X-Frame-Options: SAMEORIGIN
www.sourceforge.net X-Frame-Options: SAMEORIGIN
www.spotify.com X-Frame-Options: SAMEORIGIN
www.stackoverflow.com X-Frame-Options: SAMEORIGIN
www.tape.tv X-Frame-Options: SAMEORIGIN
www.web.de X-Frame-Options: deny
www.wikihow.com X-Frame-Options: SAMEORIGIN
www.wordpress.com X-Frame-Options: SAMEORIGIN
www.yandex.ru X-Frame-Options: DENY
www.youtube.de X-Frame-Options: SAMEORIGIN

Usage of HSTS Strict-Transport-Security

HSTS headers can only be found on a few front pages (8 of 245). Maybe it is visible more on the login pages and is avoided on front pages for performance reasons, maybe not. That would require further analysis. What can be said is only some larger technology leaders are brave enough to use it on the front page:

Website Header
www.blogger.com Strict-Transport-Security: max-age=10893354; includeSubDomains
www.blogspot.com Strict-Transport-Security: max-age=10893354; includeSubDomains
www.facebook.com Strict-Transport-Security: max-age=2592000
www.feedburner.com Strict-Transport-Security: max-age=10893354; includeSubDomains
www.github.com Strict-Transport-Security: max-age=31536000
www.paypal.com Strict-Transport-Security: max-age=14400
www.spotify.com Strict-Transport-Security: max-age=31536000
www.upjers.com Strict-Transport-Security: max-age=47336400

Conclusion

Security headers are not wide-spread on website front pages at least. Most used is the X-Frame-Option header to prevent clickjacking. Next following is X-Content-Type-Options to prevent MIME sniffing. Both of course are easy to implement as they most probably do not change your websites behaviour. I'd expect to see more HSTS on bank and other online payment service websites, but it might well be that the headers appear only on subsequent redirects when logging in, which this scan doesn't do. With CSP being the hardest to implement, as you need to have complete control over all domain usage by application content and partner content you embed, it is no wonder that only Github.com has implemented it. For me it is an indication how clean their web application actually is.

Website Technology Changes in March 2014

As in the last months let's look into changes visible at the frontend pages of the biggest websites. This time I compared the changes between February to April.

These last two months saw the usual lot of version upgrades, along with some probably unintended un-hiding of server versions, several sites going to CloudFlare as well as a premiere with IPv6 being available on the first adult movie site.

The detailed results can be found here:

What Changed?

DNS-Prefetching The HTML header based DNS prefetching is expanding once more and for the first time used on adult site: redtube.com
IPv6 An AAAA record was sighted for the first time for xhamster.com. That makes IPv6 available for the first time on a major adult site!
CDN Changes
Version Upgrades
  • dooyoo.de upgrades from PHP 5.3.2-1ubuntu4.15 to recent 4.23
  • adf.ly upgrades from PHP 5.4.21 to 5.5.8
  • duden.de upgrades from PHP 5.3.3-7+squeeze18 to 19
  • duden.de upgrades from PHP 4.4.6 to 5.4.16
  • wikipedia.de upgrades from PHP 5.3.3-7+squeeze9 to 19
  • wikipedia.org upgrades from PHP 5.3.19-1ubuntu3.9+wmf1 to 3.10+wmf1
  • xhamster.com upgrades from PHP 5.3.21 to 5.3.26
  • jquery.com upgrades from nginx 1.4.4 to 1.4.7
  • xhamster.com upgrades from nginx 1.4.4 to 1.4.7
  • qq.com upgrades from squid 3.1.18 to 3.2.1
Hiding Server Version Against the trend in the last month this month three sites have unhidden the previously hidden server details:
  • edarling.de normally not showing any server version displayed "PHP/5.4.4-14+deb7u7" in February.
  • greenpeace.de previously not indicating the Apache version now shows CentOS Apache 2.2.15
  • jamba.de previously hiding the server version in April indicated "Servlet 2.4; JBoss-4.3.0.GA_CP06 (build"

Note: the website links lead to a history page for the different sites were you can see the change details.

Caution!

All the results listed above are based on a simple scanning script. The results present a snapshot of the websites and a single response only. This is of course not necessarily an indicating for what techniques the site uses in daily operations!

Syndicate content