Cheat Sheets

Recent Posts

Helm Error: cannot connect to Tiller

Today I ran "helm" and got the following error:

$ helm status
Error: could not find tiller
It took me some minutes to find the root cause. First thing I thought was, that the tiller installation was gone/broken, which turned out to be fine. The root cause was that the helm client didn't select the correct namespace and probably stayed in the current namespace (where tiller isn't located).

This is due to the use of an environment variable $TILLER_NAMESPACE (as suggested in the setup docs) which I forgot to persist in my shell.

So running
$ TILLER_NAMESPACE=tiller helm status
solved the issue.

Using Linux keyring secrets from your scripts

When you write script that need to perform remote authentication you don't want to include passwords plain text in the script itself. And if the credentials are personal credentials you cannot deliver them with the script anyway.

libsecret

Since 2008 the Secret Service API is standardized via freedesktop.org and is implemented by GnomeKeyring and ksecretservice. Effectivly there is standard interface to access secrets on Linux desktops.

Sadly the CLI tools are rarely installed by default so you have to add them manually. On Debian
apt install libsecret-tools

Using secret-tool

There are two important modes:

Fetching passwords

The "lookup" command prints the password to STDOUT
/usr/bin/secret-tool lookup <key> <name>

Storing passwords

Note that with "store" you do not pass the password, as a dialog is raised to add it.
/usr/bin/secret-tool store <key> <name>

Scripting with secret-tool

Here is a simple example Bash script to automatically ask, store and use a secret:
#!/bin/bash

ST=/usr/bin/secret-tool LOGIN="my-login" # Unique id for your login LABEL="My special login" # Human readable label

get_password() { $ST lookup "$LOGIN" "$USER" }

password=$( get_password ) if [ "$password" = "" ]; then $ST store --label "$LABEL" "$LOGIN" "$USER" password=$( get_password ) fi

if [ "$password" = "" ]; then echo "ERROR: Failed to fetch password!" else echo "Credentials: user=$USER password=$password" fi

Note that the secret will appear in the "Login" keyring. On GNOME you can check the secret with "seahorse".

How to install Helm on Openshift

This is a short summary of things to consider when installing Helm on Openshift.

What is Helm?

Before going into details: helm is a self-proclaimed "Kubernetes Package Manager". While this is not entirly false in my opinion it is three thingsWhen looking closer it does more of the stuff that automation tools like Puppet, Chef and Ansible do.

Current Installation Issues

Since kubernetes v1.6.1, which introduced RBAC (role based access control) it became harder to properly install helm. Actually the simple installation as suggested on the homepage
# Download and...
helm init
seems to work, but as soon as you run commands like
helm list
you get permission errors. This of course being caused by the tighter access control now being in place. Sadly even now being at kubernetes 1.8 helm still wasn't updated to take care of the proper permissions.

Openshift to the rescue...

As Redhat somewhat pioneered RBAC in Openshift with their namespace based "projects" concept they are also the ones with a good solution for the helm RBAC troubles.

Setting up Helm on Openshift

Client installation (helm)

curl -s https://storage.googleapis.com/kubernetes-helm/helm-v2.6.1-linux-amd64.tar.gz | tar xz
sudo mv linux-amd64/helm /usr/local/bin
sudo chmod a+x /usr/local/bin/helm

helm init --client-only

Server installation (tiller)

With helm being the client only, Helm needs an agent named "tiller" on the kubernetes cluster. Therefore we create a project (namespace) for this agent an install it with "oc create"
export TILLER_NAMESPACE=tiller
oc new-project tiller
oc project tiller
oc process -f https://github.com/openshift/origin/raw/master/examples/helm/tiller-template.yaml -p TILLER_NAMESPACE="${TILLER_NAMESPACE}" | oc create -f -
oc rollout status deployment tiller

Preparing your projects (namespaces)

Finally you have to give tiller access to each of the namespaces you want someone to manage using helm:
export TILLER_NAMESPACE=tiller
oc project 
oc policy add-role-to-user edit "system:serviceaccount:${TILLER_NAMESPACE}:tiller"
After you did this you can deploy your first service, e.g.
helm install stable/redis --namespace 

See also Helm - Cheat Sheet kubernetes - Cheat Sheet Openshift - Cheat Sheet

Apply-changes-to-limits.conf-immediately

See also ulimit - Cheat Sheet

Sometimes you need to increase the open file limit for an application server or the maximum shared memory for your ever-growing master database. In such a case you edit your /etc/security/limits.conf and then wonder how to get the changed limits to be visible to check wether you have set them correctly. You do not want to find out that they were wrong after your master DB doesn't come up after some incident in the middle of the night...

Instant Applying Limits to Running Processes

Actually you might want to apply the changes directly to a running process additionally to changing /etc/security/limits.conf. In recent edge Linux distributions (e.g. Debian Jessie) there is a tool "prlimit" to get/set limits.

Usage for changing limits for a PID is

prlimit --pid <pid> --<limit>=<soft>:<hard>
for example
prlimit --pid 12345 --nofile=1024:2048
If you are unlucky and do not have prlimit yet check out "man 2 prlimit" for instructions on how to compile your own version because despite missing user tool the prlimit() system call is in the kernel for quite a while (since 2.6.36).

Alternative #1: Re-Login with "sudo -i"

If you do not have prlimit yet and want a changed limit configuration to become visible you might want to try "sudo -i". The reason: you need to re-login as limits from /etc/security/* are only applied on login!

But wait: what about users without login? In such a case you login as root (which might not share their limits) and sudo into the user: so no real login as the user. In this case you must ensure to use the "-i" option of sudo:
sudo -i -u <user>
to simulate an initial login with sudo. This will apply the new limits.

Alternative #2: Make it work for sudo without "-i"

Wether you need "-i" depends on the PAM configuration of your Linux distribution. If you need it then PAM probably loads "pam_limit.so" only in /etc/pam.d/login which means at login time but no on sudo. This was introduced in Ubuntu Precise for example. By adding this line

session    required   pam_limits.so
in /etc/pam.d/sudo limits will also be applied when running sudo without "-i". Still using "-i" might be easier.

Finally: Always Check Effective Limits

The best way is to change the limits and check them by running
prlimit               # for current shell
prlimit --pid <pid>   # for a running process
because it shows both soft and hard limits together. Alternatively call
ulimit -a                # for current shell
cat /proc/<pid>/limits   # for a running process
with the affected user.

Nagios Check

You might also want to have a look at the nofile limit Nagios check.

Nagios Check Plugin for nofile Limit

Following the recent post on how to investigate limit related issues which gave instructions what to check if you suspect a system limit to be hit I want to share this Nagios check to cover the open file descriptor limit. Note that existing Nagios plugins like this only check the global limit, only check one application or do not output all problems. So here is my solution which does:

  1. Check the global file descriptor limit
  2. Uses lsof to check all processes "nofile" hard limit
It has two simple parameters -w and -c to specify a percentage threshold. An example call:
./check_nofile_limit.sh -w 70 -c 85
could result in the following output indicating two problematic processes:
WARNING memcached (PID 2398) 75% of 1024 used CRITICAL apache (PID 2392) 94% of 4096 used
Here is the check script doing this:
#!/bin/bash


# Check "nofile" limit for all running processes using lsof

MIN_COUNT=0 # default "nofile" limit is usually 1024, so no checking for # processes with much less open fds needed

WARN_THRESHOLD=80 # default warning: 80% of file limit used CRITICAL_THRESHOLD=90 # default critical: 90% of file limit used

while getopts "hw:c:" option; do case $option in w) WARN_THRESHOLD=$OPTARG;; c) CRITICAL_THRESHOLD=$OPTARG;; h) echo "Syntax: $0 [-w <warning percentage>] [-c <critical percentage>]"; exit 1;; esac done

results=$( # Check global limit global_max=$(cat /proc/sys/fs/file-nr 2>&1 |cut -f 3) global_cur=$(cat /proc/sys/fs/file-nr 2>&1 |cut -f 1) ratio=$(( $global_cur * 100 / $global_max))

if [ $ratio -ge $CRITICAL_THRESHOLD ]; then echo "CRITICAL global file usage $ratio% of $global_max used" elif [ $ratio -ge $WARN_THRESHOLD ]; then echo "WARNING global file usage $ratio% of $global_max used" fi

# We use the following lsof options: # # -n to avoid resolving network names # -b to avoid kernel locks # -w to avoid warnings caused by -b # +c15 to get somewhat longer process names # lsof -wbn +c15 2>/dev/null | awk '{print $1,$2}' | sort | uniq -c |\ while read count name pid remainder; do # Never check anything above a sane minimum if [ $count -gt $MIN_COUNT ]; then # Extract the hard limit from /proc limit=$(cat /proc/$pid/limits 2>/dev/null| grep 'open files' | awk '{print $5}')

# Check if we got something, if not the process must have terminated if [ "$limit" != "" ]; then ratio=$(( $count * 100 / $limit )) if [ $ratio -ge $CRITICAL_THRESHOLD ]; then echo "CRITICAL $name (PID $pid) $ratio% of $limit used" elif [ $ratio -ge $WARN_THRESHOLD ]; then echo "WARNING $name (PID $pid) $ratio% of $limit used" fi fi fi done )

if echo $results | grep CRITICAL; then exit 2 fi if echo $results | grep WARNING; then exit 1 fi

echo "All processes are fine."
Use the script with caution! At the moment it has no protection against a hanging lsof. So the script might mess up your system if it hangs for some reason. If you have ideas how to improve it please share them in the comments!

Solving rtl8812au installation on Ubuntu 17.10

After several fruitless attempts on getting my new dual band Wifi stick to work on my PC I went the hard way to compiling the driver.

Figuring out which driver to use

As drivers do support device ids the first thing is to determine the id
$ lsusb
[...]
Bus 002 Device 013: ID 0bda:a811 Realtek Semiconductor Corp. 
So the id being "0bda:a811" you can search online for a list of driver names. Google suggests rtl8812au as related searches...

Finding a source repo

At github you can find several source repos for the rtl8812au driver in of different age. It seems that Realtek is supplying the source on some driver CDs and different people independently put them online. The hard part is to find the most recent one, as only this has fixes for recent kernels.

One with patches for kernel 4.13.x is the one from Vital Koshalev which has already a PR against the most commonly referenced repo of diederikdehaas which doesn't work yet!.

Getting the right source

So fetch the source as following
git clone https://github.com/VitalKoshalew/rtl8812AU.git
git checkout driver-4.3.22-beta-mod

Compilation + Installation of rtl8812AU

The instructions are from the README.md and are to be run as root:
DRV_NAME=rtl8812AU
DRV_VERSION=4.3.22-beta-mod
mkdir /usr/src/${DRV_NAME}-${DRV_VERSION}
git archive driver-${DRV_VERSION} | tar -x -C /usr/src/${DRV_NAME}-${DRV_VERSION}
dkms add -m ${DRV_NAME} -v ${DRV_VERSION}
dkms build -m ${DRV_NAME} -v ${DRV_VERSION}
dkms install -m ${DRV_NAME} -v ${DRV_VERSION}
See also DKMS - Cheat Sheet

Loading rtl8812AU

If everything worked well you should now be able to issue
modprobe 8812au
Note the missing "rtl"!!!

If the module loads, but the wifi stick doesn't work immediately it might be that the rtlwifi driver is preventing the self-compiled module from working. So remove it with
rmmod rtlwifi
It will complain about dependencies. You need to rmmod those too. Afterwards the new driver should load properly. To make disabling rtlwifi persistent add it to the modprobe blacklist:
echo "blacklist rtlwifi" >>/etc/modprobe.d/blacklist.conf
Please leave feedback on the instructions if you have problems!

Ensure secure Javascript dependencies

When you write Javascript code or when you want to know if a 3rd party code bases dependencies are secure check out https://david-dm.org which is an online scanner for github repos package.json contents.

This tool is able to generate badges and gives you details on dependencies

Here is a screenshot of some vulnerable deps



and the badge as seen on the corresponding github page:



While I do not like the badge explosion on github.com it still is an amazingly useful tool to know the issue with this library just looking at the github project.

Openshift S2I and Spring profiles

When porting Springboot applications to Openshift using S2I (source to image) directly from a git repo you cannot rely on a start script passing the proper -Dspring.profiles.active=<profile name> parameter like this

java -Dspring.profiles.active=development -jar yourApplication.jar
The only proper ways for injecting application configuration are
  1. Add it to the artifact/repository (meeh)
  2. Mount it using a config map
  3. Pass it via Docker environment variables
And for Openshift variant #3 works fine as there is an environment variable SPRING_PROFILES_ACTIVE which you can add in your deployment configuration and set it to your favourite spring profile name.

USB seq nnnn is taking a long time

When your dmesg/journalctl/syslog says something like

Nov 14 21:43:12 Wolf systemd-udevd[274]: seq 3634 '/devices/pci0000:00/0000:00:14.0/usb2/2-1' is taking a long time
then know that the only proper manly response can be
systemctl restart udev
Don't allow for disrespectful USB messages!!!

Openshift Ultra-fast Bootstrap

Today I want to share some hints on ultra-fast bootstrapping developers to use Openshift. Given that adoption in your organisation depends on developers daring and wanting to use Kubernetes/Openshift I believe showing a clear and easy migration path is the way to go.

Teach the Basics by Failing!

Actually why not treating Openshift as a user-friendly self-service? Naively approach it and try stuff.

So hold a workshop. Ask people to:
  1. Not use the CLI for now!!! Don't even think about it. Automation comes later!
  2. Login and create a project. That usually works well.
  3. Decide on Docker Image / Source to Image. In the second dialog of the project creation you get presented with those three tabs

    Let them choose their poison.

    Let the image pull fail because they don't find docker images and don't know that they cannot just fetch stuff from docker.com. Explain why this is the case. Show them where to find your preferred base and runtime images in you internal registry which of course is already configured, ready to be used. Show them the base image you suggest.

    Let the template creation fail using an already prepared template. Show were to look up the build error and explain where to find the infamously hidden secrets option everyone needs.

  4. Once the first build fails: explain the logic of applications in Openshift and that they did not only create a project, but also an application. Show the difference of locating builds and deployments. Show how to access logs of both and how to find 'Edit' hidden in the 'Actions' menu.
  5. When the build fails du to SSH connection refused: Explain that (even when using a source secret you already prepared) you need to put the public key in your favourite SCM either globally, per project or per repo for the code pull to work.
  6. When people check the pod first and see it isn't running: Explain again and again the holy trinity of checking stuff:
    1. First check the build
    2. Then check the deployment
    3. Only then check if pods do come up
  7. Finally the pod is green! People will access the deployed application and ask you how? Now is the time to have a short excurse on service and routing. Maybe show an already configured defautl. If some service isn't accessible:
    1. Show how to get the pods TCP endpoint
    2. Show how to attach to a container via the GUI / CLI

Have Docs and Examples Ready

Most important of course is preparation. Do prepare
  1. Walkthrough screenshots
  2. At least one runtime template
  3. At least one base image on your own registry
  4. At least one S2I ready source repository with a hello-world app
  5. Global project settings with
    • a default SSH key for source pulling
    • access configured for your own docker registry
  6. Maybe make an example project visible for all newbies

Things to avoid...

This is of course quite opinionated, but think about it:

Finally...

Be prepared to iterate this again and again as often as needed.

Docker disable ext4 journaling

Noteworthy point from the Remind Ops: when running docker containers on ext4 consider disabling journaling. Why, because a throw-away almost read-only filesystem doesn't need recovery on crash.

See also Docker - Cheat Sheet

Gedit ShellCheck Linter Plugin

Today I had enough of the missing shell linting support in Gedit. So I took the time and derived a gedit-shellcheck plugin from an already existing JSHint plugin written by Xavier Gendre.

Usage

The linter can be run using Ctrl-J or from the Tools menu 'Check with ShellCheck'. Here is a screenshot

Setup

To use the plugin you need ShellCheck
apt install shellcheck
and you need to place the plugin in your Gedit plugins folder.
git clone https://github.com/lwindolf/gedit-shellcheck.git
mkdir -p ~/.local/share/gedit/plugins/
cp -r gedit-shellcheck/shellcheck.plugin gedit-shellcheck/shellcheck/ ~/.local/share/gedit/plugins/
Finally restart Gedit and activate the plugin.

Python re.sub Examples

Example for re.sub() usage in Python

Syntax

import re

result = re.sub(pattern, repl, string, count=0, flags=0);

Simple Examples

num = re.sub(r'abc', '', input)              # Delete pattern abc
num = re.sub(r'abc', 'def', input)           # Replace pattern abc -> def
num = re.sub(r'\s+', '\s', input)            # Eliminate duplicate whitespaces
num = re.sub(r'abc(def)ghi', '\1', input)    # Replace a string with a part of itself

Advance Usage

Replacement Function

Instead of a replacement string you can provide a function performing dynamic replacements based on the match string like this:
def my_replace(m):
    if :
       return <replacement variant 1>
    return <replacement variant 2>


result = re.sub("\w+", my_replace, input)

Count Replacements

When you want to know how many replacements did happen use re.subn() instead:
result = re.sub(pattern, replacement, input)
print ('Result: ', result[0])
print ('Replacements: ', result[1])

See also: Python - Cheat Sheet Python re.match - Cheat Sheet Python re.sub - Cheat Sheet

The silent death of .xprofile in GNOME 3.22

Did you upgrade to Wayland already? Noticed breaking stuff everywhere?

One of the silent breaks is .xprofile, Wayland by design doesn't load the X legacy initialisation scripts ~/.xprofile ~/.xsession ~/.xinitrc as well as their siblings below /etc/X11.

While I understand the reason to cut away legacy to forward a modern desktop environment, I'd still like GNOME to care more about helping end-users to migrate.

How to Migrate Environment Settings

If you want to set desktop global environment variables you need to task your display manager which those as Wayland explicitely refuses to provide a hook. When you use GDM a possible hook is in /usr/share/gdm/env.d/startup_${USER}.sh

How to Migrate Application Startup

The only sane way is to migrate to the autostart as defined by the XDG spec which means placing .desktop files in (usually) ~/.config/autostart.

To run an arbitrary command use a shell invocation like this
[Desktop Entry]
Name=My favourite command
GenericName=My favourite command
Comment=We all like GNOME making life harder
Exec=/bin/bash -c "ENVVAR=value my_application -o --param=value something"
Terminal=false
StartupNotify=true
Type=Application
Categories=X11;Legacy;Startup
The only relevant line being the 'Exec' entry. Use a shell with -c to pass environment variables or more complex commands.

Finally...

Maybe it is a good moment to look back at the long cross-platform compatibility ending with X11 being replaced. It was great the apply old Unix knowledge on every Linux desktop so far. Now simply stuff like xkill and xprops just won't work anymore. Don't even think about positioning windows at certain coordinates.

Time to learn new Wayland-only commands.

How to find the process listening on a given port

Use one of the following commands to resolve an open port to a running process.

How to use lsof

lsof -i :80

Alternative using netstat

netstat -tlnp | grep :80

Alternative using ss

Using ss has the advantage of given the full process name (not truncated like netstat) and also prints all PIDs having the port open, as lsof does.
ss -tlnp | grep :80

Puppet Dry Run

To do a "dry run" in Puppet you need to invoke the agent in noop mode:

puppet agent -t --noop

Limited Dry Run

If you don't want a full run, but check specific ressources/classes/... you can also invoke a dry-run for a tag like this:
puppet agent -t --noop --tags Cron
Which would show all potential changes for all Cron ressources.

Drift Logging using Noop Runs

When you do not do periodic runs pulling all new configuration to all your servers. You might want to do period noop runs instead. This is because like normal runs, noop runs also generated reports against the reporting server (e.g. Foreman or PuppetDB). This allows you to view statistics on noop ressources in your reporting server. The Foreman dashboard for example has a good pie chart for visualizing configuration drift on your servers.

Noop from the code

You can utilize the noop mode even in the code: noop is a meta parameter that can be applied to resources and types to avoid them to actually do something. So declaring
file { '/data/my_favourite_file.txt':
   ensure => absent,
   noop   => true
}
will raise a noop event on all affected systems, that Puppet wants to remove the file, but didn't because of the noop flag.

So imagine a critical change you want to put live. Yes, of course you have tested it, but you can make extra sure by

HowTo Mount LVM Partitions

Find out which LVM parititions you have by running

lvdisplay
and mount the one you need with
mount /dev/vg0/vol1 /mnt
See also: LVM - Cheat Sheet