Cheat Sheets

Recent Posts

USB seq nnnn is taking a long time

When your dmesg/journalctl/syslog says something like

Nov 14 21:43:12 Wolf systemd-udevd[274]: seq 3634 '/devices/pci0000:00/0000:00:14.0/usb2/2-1' is taking a long time
then know that the only proper manly response can be
systemctl restart udev
Don't allow for disrespectful USB messages!!!

Openshift Ultra-fast Bootstrap

Today I want to share some hints on ultra-fast bootstrapping developers to use Openshift. Given that adoption in your organisation depends on developers daring and wanting to use Kubernetes/Openshift I believe showing a clear and easy migration path is the way to go.

Teach the Basics by Failing!

Actually why not treating Openshift as a user-friendly self-service? Naively approach it and try stuff.

So hold a workshop. Ask people to:
  1. Not use the CLI for now!!! Don't even think about it. Automation comes later!
  2. Login and create a project. That usually works well.
  3. Decide on Docker Image / Source to Image. In the second dialog of the project creation you get presented with those three tabs

    Let them choose their poison.

    Let the image pull fail because they don't find docker images and don't know that they cannot just fetch stuff from docker.com. Explain why this is the case. Show them where to find your preferred base and runtime images in you internal registry which of course is already configured, ready to be used. Show them the base image you suggest.

    Let the template creation fail using an already prepared template. Show were to look up the build error and explain where to find the infamously hidden secrets option everyone needs.

  4. Once the first build fails: explain the logic of applications in Openshift and that they did not only create a project, but also an application. Show the difference of locating builds and deployments. Show how to access logs of both and how to find 'Edit' hidden in the 'Actions' menu.
  5. When the build fails du to SSH connection refused: Explain that (even when using a source secret you already prepared) you need to put the public key in your favourite SCM either globally, per project or per repo for the code pull to work.
  6. When people check the pod first and see it isn't running: Explain again and again the holy trinity of checking stuff:
    1. First check the build
    2. Then check the deployment
    3. Only then check if pods do come up
  7. Finally the pod is green! People will access the deployed application and ask you how? Now is the time to have a short excurse on service and routing. Maybe show an already configured defautl. If some service isn't accessible:
    1. Show how to get the pods TCP endpoint
    2. Show how to attach to a container via the GUI / CLI

Have Docs and Examples Ready

Most important of course is preparation. Do prepare
  1. Walkthrough screenshots
  2. At least one runtime template
  3. At least one base image on your own registry
  4. At least one S2I ready source repository with a hello-world app
  5. Global project settings with
    • a default SSH key for source pulling
    • access configured for your own docker registry
  6. Maybe make an example project visible for all newbies

Things to avoid...

This is of course quite opinionated, but think about it:

Finally...

Be prepared to iterate this again and again as often as needed.

Docker disable ext4 journaling

Noteworthy point from the Remind Ops: when running docker containers on ext4 consider disabling journaling. Why, because a throw-away almost read-only filesystem doesn't need recovery on crash.

See also Docker - Cheat Sheet

Gedit ShellCheck Linter Plugin

Today I had enough of the missing shell linting support in Gedit. So I took the time and derived a gedit-shellcheck plugin from an already existing JSHint plugin written by Xavier Gendre.

Usage

The linter can be run using Ctrl-J or from the Tools menu 'Check with ShellCheck'. Here is a screenshot

Setup

To use the plugin you need ShellCheck
apt install shellcheck
and you need to place the plugin in your Gedit plugins folder.
git clone https://github.com/lwindolf/gedit-shellcheck.git
mkdir -p ~/.local/share/gedit/plugins/
cp -r gedit-shellcheck/shellcheck.plugin gedit-shellcheck/shellcheck/ ~/.local/share/gedit/plugins/
Finally restart Gedit and activate the plugin.

Python re.sub Examples

Example for re.sub() usage in Python

Syntax

import re

result = re.sub(pattern, repl, string, count=0, flags=0);

Simple Examples

num = re.sub(r'abc', '', input)              # Delete pattern abc
num = re.sub(r'abc', 'def', input)           # Replace pattern abc -> def
num = re.sub(r'\s+', '\s', input)            # Eliminate duplicate whitespaces
num = re.sub(r'abc(def)ghi', '\1', input)    # Replace a string with a part of itself

Advance Usage

Replacement Function

Instead of a replacement string you can provide a function performing dynamic replacements based on the match string like this:
def my_replace(m):
    if :
       return <replacement variant 1>
    return <replacement variant 2>


result = re.sub("\w+", my_replace, input)

Count Replacements

When you want to know how many replacements did happen use re.subn() instead:
result = re.sub(pattern, replacement, input)
print ('Result: ', result[0])
print ('Replacements: ', result[1])

See also: Python - Cheat Sheet Python re.match - Cheat Sheet Python re.sub - Cheat Sheet

The silent death of .xprofile in GNOME 3.22

Did you upgrade to Wayland already? Noticed breaking stuff everywhere?

One of the silent breaks is .xprofile, Wayland by design doesn't load the X legacy initialisation scripts ~/.xprofile ~/.xsession ~/.xinitrc as well as their siblings below /etc/X11.

While I understand the reason to cut away legacy to forward a modern desktop environment, I'd still like GNOME to care more about helping end-users to migrate.

How to Migrate Environment Settings

If you want to set desktop global environment variables you need to task your display manager which those as Wayland explicitely refuses to provide a hook. When you use GDM a possible hook is in /usr/share/gdm/env.d/startup_${USER}.sh

How to Migrate Application Startup

The only sane way is to migrate to the autostart as defined by the XDG spec which means placing .desktop files in (usually) ~/.config/autostart.

To run an arbitrary command use a shell invocation like this
[Desktop Entry]
Name=My favourite command
GenericName=My favourite command
Comment=We all like GNOME making life harder
Exec=/bin/bash -c "ENVVAR=value my_application -o --param=value something"
Terminal=false
StartupNotify=true
Type=Application
Categories=X11;Legacy;Startup
The only relevant line being the 'Exec' entry. Use a shell with -c to pass environment variables or more complex commands.

Finally...

Maybe it is a good moment to look back at the long cross-platform compatibility ending with X11 being replaced. It was great the apply old Unix knowledge on every Linux desktop so far. Now simply stuff like xkill and xprops just won't work anymore. Don't even think about positioning windows at certain coordinates.

Time to learn new Wayland-only commands.

How to find the process listening on a given port

Use one of the following commands to resolve an open port to a running process.

How to use lsof

lsof -i :80

Alternative using netstat

netstat -tlnp | grep :80

Alternative using ss

Using ss has the advantage of given the full process name (not truncated like netstat) and also prints all PIDs having the port open, as lsof does.
ss -tlnp | grep :80

Puppet Dry Run

To do a "dry run" in Puppet you need to invoke the agent in noop mode:

puppet agent -t --noop

Limited Dry Run

If you don't want a full run, but check specific ressources/classes/... you can also invoke a dry-run for a tag like this:
puppet agent -t --noop --tags Cron
Which would show all potential changes for all Cron ressources.

Drift Logging using Noop Runs

When you do not do periodic runs pulling all new configuration to all your servers. You might want to do period noop runs instead. This is because like normal runs, noop runs also generated reports against the reporting server (e.g. Foreman or PuppetDB). This allows you to view statistics on noop ressources in your reporting server. The Foreman dashboard for example has a good pie chart for visualizing configuration drift on your servers.

Noop from the code

You can utilize the noop mode even in the code: noop is a meta parameter that can be applied to resources and types to avoid them to actually do something. So declaring
file { '/data/my_favourite_file.txt':
   ensure => absent,
   noop   => true
}
will raise a noop event on all affected systems, that Puppet wants to remove the file, but didn't because of the noop flag.

So imagine a critical change you want to put live. Yes, of course you have tested it, but you can make extra sure by

HowTo Mount LVM Partitions

Find out which LVM parititions you have by running

lvdisplay
and mount the one you need with
mount /dev/vg0/vol1 /mnt
See also: LVM - Cheat Sheet

Is there hope when your Couchbase cluster is stuck in compacting buckets?

Well to be anticlimactic: no.

Scope

This seems to be at least a Couchbase 3.x problem. So far I haven't experienced it with Couchbase 4. Of both versions I only know about the so called community edition.

As for the frequency: Couchbase 3 getting stuck on bucket compacting is propabilistic. In the setups I've run so far it happens every half a year. But this might be load-dependant. Actually never having had the issue on some "smaller" clusters, I actually think it is.

The Symptoms

If you do not monitor explicitly for the compacting status, you will probably noticy by some nodes disks running full. Compacting not working anymore means, the Couchbase disk fragmentation growing and finally filling you disks.

If you look in the GUI you will see a constant "Compacting..." indicator in the top right of the admin GUI. In normal operation it never takes more than some minutes to finish (again depending on your usage).

Things that do not work...

What does help...

The root cause

What actually happened is a data structure corruption from which Couchbase 3 does not recover. This is also the reason why flushing buckets helps.

There are several bug reports in Couchbase 2, 3 and 4 about compacting stuck for different reasons. In general Couchbase is not a very stable product in this regard...

How to search Confluence for macro usage

When you want to find all pages in Confluence that embed a certain macro you cannot simply use the search field as it seamily only searches the resulting content. A normal search query does not check the markup for the macro code.

To search for a certain macro do a request like this

https://<base url>/dosearchsite.action?cql=macro+%3D+"<macro name>"
So to search for the "sql-query" macro for example do
https://<base url>/dosearchsite.action?cql=macro+%3D+"sql-query"

Solving d3.scale is undefined

When porting older code and examples of d3.js visualizations you might encounter the following exception:

TypeError: d3.scale is undefined
The causing code might be something like:
var xscale = d3.scale.linear().range([0, chartWidth]);
The problem is an API change from d3.scale.linear() to d3.scaleLinear() with d3.js 4.0. So to fix it rewrite the code with
var xscale = d3.scaleLinear().range([0, chartWidth]);

Automatically Download Oracle JDK

When downloading Oracle JDK via scripts you might run into the login page.

While in the past it was sufficient to do something like

wget --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
you now also need to ignore https:// as you will get redirected. So when using wget you might want to run
wget -c --no-cookies --no-check-certificate --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
Note the added "-c" and "--no-check-certificate" options.

RabbitMQ Does Not Start: init terminating in do_boot

If you have a RabbitMQ cluster and a crashed node fails to start again with

{"init terminating in do_boot",{undef,[{rabbit_prelaunch,start,[]},{init,start_it,1},{init,start_em,1}]}}
in /var/log/rabbitmq/startup_log and something like
Error description:
   {could_not_start,rabbitmq_management,
       {{shutdown,
            {failed_to_start_child,rabbit_mgmt_sup,
                {'EXIT',
                    {{shutdown,
                         [{{already_started,<9180.461.0>},
                           {child,undefined,rabbit_mgmt_db,
                               {rabbit_mgmt_db,start_link,[]},
                               permanent,4294967295,worker,
                               [rabbit_mgmt_db]}}]},
                     {gen_server2,call,
                         [<0.427.0>,{init,<0.425.0>},infinity]}}}}},
        {rabbit_mgmt_app,start,[normal,[]]}}}

Log files (may contain more information): /var/log/rabbitmq/[email protected] /var/log/rabbitmq/[email protected]
in /var/log/rabbitmq/[email protected] then you might want to try to drop the node from the cluster by running
rabbitmqctl forget_cluster_node [email protected]
one a working cluster node and rejoin the node by running
rabbitmqctl join_cluster [email protected]
on the disconnected node (given rabbit-02 is a working cluster member).

Note: Doing this might make you lose messages!

Match Structured Facts in MCollective

If you are using Facter 2+, which is what you do when you run at least Puppet4, then you have structured facts (meaning nested values) like those:

processors => {
  count => 2,
  isa => "unknown",
  models => [
    "QEMU Virtual CPU version 2.1.2",
    "QEMU Virtual CPU version 2.1.2"
  ],
  physicalcount => 2
}
Now you cannot match those using
mco find -F <fact name>=<fact value>
If you try you just get an empty result. The only way to match structured facts is using -S
mco find -S 'fact("<fact name>").value=<value>'
For example:
mco find -S 'fact("networking.network").value=192.168.5.0'
mco find -S 'fact("os.distro.codename").value=jessie'
See also Mcollective Cheat Sheet

Nagios Check for Systemd Failed Units

Just a short bash script to check for faulty systemd units to avoid 107 lines of Python...

#!/bin/bash

if [ -f /bin/systemctl ]; then failed=$(/bin/systemctl --failed --no-legend) failed=${failed/ */} # Strip everything after first space failed=${failed/.service/} # Strip .service suffix

if [ "$failed" != "" ]; then echo "Failed units: $failed" exit 1 else echo "No failed units." exit 0 fi else echo "No systemd. Nothing was checked!" exit 0 fi

How to fix debsecan for Wheezy

Someone at Debian increase security for all Debian servers by breaking debsecan a while ago for everything before Jessie by moving the vulnerability definitions from

http://secure-testing.debian.net/debian-secure-testing/project/debsecan/release/1/

to

https://security-tracker.debian.org/tracker/debsecan/release/1

Of course there was no way to issue a security fix for Wheezy debsecan...

Workaround 1: Hotfix

So if you still want to scan your Wheezy systems you can hotfix debsecan before running it like this:
sed -i "s/http:\/\/secure-testing.debian.net\/debian-secure-testing/https:\/\/security-tracker.debian.org\/tracker/;s/project\/debsecan\/release\/1\//debsecan\/release\/1\//" /usr/bin/debsecan 

Workaround 2: Pass Config

You can also pass an in-place config file:
debsecan --config <(echo SOURCE="https://security-tracker.debian.org/tracker/debsecan/release/1/")