Linux sysadmin links

echo 1 > /proc/sys/vm/block_dump

# wait some time...
echo 0 > /proc/sys/vm/block_dump

# Now check syslog for block dump lines

dmesg -T      # Enable human readable timestamps
dmesg -x      # Show facility and log level
dmesg -f daemon     # Filter for facility daemon
dmesg -l err,crit,alert,emerg # Filter for errors

lslk - Find file locks: Use lslk to find which PID is blocking an flock() to a file.

lsof - Find owners of open file handles:

lsof      # Complete list
lsof -i :22    # Filter single TCP port
lsof [email protected]:22 # Filter single connection endpoint
lsof -u <user>   # Filter per user
lsof -c <name>   # Filter per process name
lsof -p 12345    # Filter by PID
lsof /etc/hosts   # Filter single file

Perf Tutorial: 2.6+ generic kernel performance statistics tool. perf stat -B some_command
dstat: Replaces vmstat, iostat, netstat and ifstat and allows to determine PID that is most CPU and most I/O expensive dstat -a --top-bio --top-cpu
iotop: Python script to monitor I/O like top
PHP - How to setup the APD debugger

Filesystem / Partitioning

uNetBootin: Create bootable media for any distribution. Most useful with USB sticks.
Convert ext2 to ext3: tune2fs -j /dev/hda1
Convert ext3 to ext4: tune2fs -O extents,uninit_bg,dir_index /dev/sda1
Determine Inode Count: tune2fs -l /dev/sda1 | grep Inode
Disable ext4 barriers: Add "barrier=0" to the mount options.

LVM - Add another disk: How to add a disk to an existing volume

# Setup partition with (use parted for >2TB)
(parted) mklabel gpt       # only when >2TB
(parted) mkpart primary lvm 0 4T    # setup disk full size (e.g. 4TB)

pvcreate /dev/sdb1       # Create physical LVM disk
vgextend vg01 /dev/sdb1      # Add to volume group
vgextend -L +4t /dev/mapper/vg01-lvdata  # Extend your volume 
resize2fs /dev/mapper/vg01-lvdata   # Auto-resize file system

rsync - --delete doesn't work: How to debug this.

Mail

postsuper -d ALL
postsuper -d ALL deferred

Middleware

Heartbeat - Manual IP Failover

# Either run on the node that should take over
/usr/share/heartbeat/hb_failover

# Or run on the node to should stop working
/usr/share/heartbeat/hb_standby

Pacemaker - Setup Steps

RabbitMQ - Commands

rabbitmqctl list_vhosts   # List all defined vhosts
rabbitmqctl list_queues <vhost> # List all queues for the vhost

rabbitmqctl report    # Dump detailed report on RabbitMQ instance

RabbitMQ - Fix Chef 100% CPU usage
RabbitMQ - Setup Clustering

Monitoring

Munin - Test Plugins:

/usr/sbin/munin-run <plugin name> # for values
/usr/sbin/munin-run <plugin name> config # for configuration

Munin - Manual Update Run:
```
sudo -u munin /usr/bin/munin-cron
```

bash - Control History Handling:

/usr/sbin/munin-node-configure --suggest

# and enable them using
/usr/sbin/munin-node-configure --shell | sh

Network

NFS - Tuning Secrets: SGI Slides on NFS Performance

nttcp - TCP performance testing

# On sending host
nttcp -t -s

# On receiving host
nttcp -r -s

tcpdump - Be verbose and print full package hex dumps:

 tcpdump -i eth0 -nN -vvv -xX -s 1500 port <some port>

SNMP - Dump all MIBs: When you need to find the MIB for an object known only by name try snmpwalk -c public -v 1 -O s <myhost> .iso | grep <search string>
Hurricane Electric - BGP Tools: Statistics on all AS as well as links to their looking glasses.

Package Management

Debian

apt-get install <package> 
apt-get remove <package> # Remove files installed by <package>
apt-get purge <package>  # Remove <package> and all the files it did create

apt-get upgrade    # Upgrade all packages
apt-get install <package> # Upgrade an install package

apt-get dist-upgrade  # Upgrade distribution

apt-cache search <package> # Check if there is such a package name in the repos
apt-cache clean    # Remove all downloaded .debs

dpkg -l      # List all installed/known packages

# More dpkg invocations above in the "Debian" section!

Ubuntu (like Debian) with the addition of

do-release-upgrade   # For Ubuntu release upgrades

OpenSuSE

zypper install <package> 

zypper refresh    # Update repository infos

zypper list-updates
zypper repos    # List configured repositories

zypper dist-upgrade   # Upgrade distribution
zypper dup     # Upgrade distribution (alias)

zypper search <package>  # Search for <package>
zypper search --search-descriptions <package>

zypper clean      # Clean package cache

# For safe updates:
zypper mr –keep-packages –remote # Enable caching of packages
zypper dup -D      # Fetch packages using a dry run
zypper mr –all –no-refresh  # Set cache usage for following dup
zypper dup      # Upgrade!

Redhat:
```
up2date
```

Centos:

yum update     # Upgrade distro
yum install <package>  # Install <package>

RAID

mdadm - Commands

cat /proc/mdstat   # Print status

mdadm --detail /dev/md0  # Print status per md

mdadm --manage -r /dev/md0 /dev/sda1 # Remove a disk
mdadm --zero-superblock /dev/sda1  # Initialize a disk
mdadm --manage -a /dev/md0 /dev/sda1 # Add a disk

mdadm --manage --set-faulty /dev/md0 /dev/sda1

hpacucli - Commands

# Show status of all arrays on all controllers
hpacucli all show config
hpacucli all show config detail

# Show status of specific controller
hpacucli ctrl=0 pd all show

# Show Smart Array status
hpacucli all show status

LSI MegaRAID - Commands

# Get number of controllers
/opt/MegaRAID/MegaCli/MegaCli64 -adpCount -NoLog

# Get number of logical drives on controller #0
/opt/MegaRAID/MegaCli/MegaCli64 -LdGetNum -a0 -NoLog

# Get info on logical drive #0 on controller #0
/opt/MegaRAID/MegaCli/MegaCli64 -LdInfo -L0 -a0 -NoLog

Security

chroot Best Practices
Debian - Package Vulnerability Scanner:
```
debsecan --suite=sid
```
FreeBSD - Package Vulnerability Scanner:
```
portaudit -Fda
```
Overview of Automated Linux Package Vulnerability Scanning
Debug Apparmor: How to determine which permissions prevent some type of access.
OpenVAS - Security Scanner (commercial)
TCP/IP Stack Hardening: Unix network hardening
OpenSSL Kurzreferenz: All commands to create keys, certificates and certificate requests.
Print x509 Certificate Infos: openssl x509 -text -in my.crt By replacing "x509" with "ca" or "crt" you can dump other file types too.
mkpasswd - Create Passwords

Shell

date: Convert To Unix Timestamp: date -d "$date" +%s
date: Convert From Unix Timestamp: date -d "1970-01-01 1234567890 sec GMT"
date: Calculate Last Day of Month: cal $(date "+%M %y") | grep -v ^$ | tail -1 | sed 's/^.* $[0-9]*$$/\1/'
bash: Extend Completion: How to setup your own bash completion schemas. complete -W 'add branch checkout clone commit diff grep init log merge mv pull push rebase rm show status tag' git
bash - kill all childs on exit: trap true TERM kill -- -$$

unset HISTFILE      # Stop logging history in this bash instance
HISTIGNORE="[ ]*"      # Do not log commands with leading spaces
HISTIGNORE="&"      # Do not log a command multiple times

HISTTIMEFORMAT="%Y-%m-%d %H:%M:%S" # Log with timestamps

bash - apply /etc/security/limits.conf change immediately: sudo -i -u <user>
Mail Attachments: Dozens of variants to mail attachments using Unix tools.
tail -f until removed: When you want to tail a file until it gets removed tail --follow=name myfile

join - DB-like joining of CSV files:

join -o1.2,2.3 -t ";" -1 1 -2 2 employee.csv tasks.csv

shell - list all commands: compgen -c |sort -u
shell - Check for interactive terminal: Run "tty" in silent mode and check the exit code tty -s
shell - ANSI color matrix
Sorting column: Use the -k switch of "sort" to sort lines by a column. E.g.
```
cat access.log | sort -k 1
```
watch: wait for file/directory changes and run a command watch -d ls -l

Shell - Unbuffer Output:

stdbuf -i0 -o0 -e0 <some command>  # Best solution

unbuffer <some command>     # Wrapper script from expect

dos2unix with vi:
```
:%s/^V^M//g
```

SSH

authorized_keys HowTo: Syntax and options...
Easy Key Copying: Stop editing authorized_keys remote. Use the standard OpenSSH ssh-copy-id instead.
```
ssh-copy-id [-i keyfile] user@maschine
```
ProxyCommand: Run SSH over a gateway and forward to other hosts based and/or perform some type of authentication. In .ssh/config you can have:
```
Host unreachable_host
  ProxyCommand ssh gateway_host exec nc %h %p
```
Transparent Multi-Hop: ssh host1 -A -t host2 -A -t host3 ...
100% non-interactive SSH: What parameters to use to avoid any interaction. ssh -i my_priv_key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PreferredAuthentications=publickey user@host -n "/bin/ls"
SFTP chroot with umask: How to enforce a umask with SFTP Subsystem sftp /usr/libexec/openssh/sftp-server -u 0002
Agent Forwarding explained with pictures! Configured in /etc/ssh_config with
```
Host *
ForwardAgent yes
```
How to use a SOCKS Proxy On the client start proxy by
```
ssh -D <port> <remote host>
```

Webserver Stack

Apache vs. Lighttpd Rewrite Rules: How to migrate.

Apache - Rewrite on File Pattern

RewriteCond %{REQUEST_FILENAME} (.*)\.(html|htm)$

Apache - Rewrite on User Agent

RewriteCond %{HTTP_USER_AGENT} (iPhone|iPad)

Apache - Exception Hook: Since 2.0.49 Apache has an exception hook to handle crashes.
```
EnableExceptionHook on
```
memcached - List of Monitoring Solutions
memcached - List of Telnet Commands

stats cachedump <slab class> <number of keys to dump>

memcached - Key Dump Tool "memdump"
memcached - Persistent Hashing: How to survive server failures without the whole cluster degrading. Of course applies to any cache.

nginx - Complex Conditions: As nginx does not support complex logic in if() conditions you need to set flags in a smart way to workaround it.

# Define a control flag
set $extra_handling = 0;

# Set the control flag when needed
if ($variable1 ~* pattern) {
    set $extra_handling = 1;
}

# Unset the flag if needed
if ( $variable2 = 1 ) {
    set $extra_handling = 0;
}

if ( $extra_handling = 1 ) {
    # Trigger intended behaviour
}

Redis - Most Important Commands for Sysadmins

This is a list of non-trivial Linux administration commands and can be used as a cheat sheet or link collection. If you find errors or want to add something please post a comment below!

Automation Products

Which automation tools are actually out there?

Bcfg2: Alternative to puppet and cfengine by Argonne National Laboratory. (IMO out-dated)
cfengine (active, commercially backed, large user base)
Chef: Alternative to puppet (Ruby, active, commercially backed, large user base)
JuJu: mostly for Ubuntu, service orchestration tool (Python, commercially backed)
Puppet (Ruby-like + Ruby, active, commercially backed, large user base)
slaughter (Perl, active, small user base)
Sprinkle (Ruby, quite recent)
Wikipedia Comparison Chart: Check here for other less known and new tools!

Automation

Augeas: Very flexible file editor to be used with Puppet or standalone. Could also work with cfengine.
```
$ augtool
augtool> set /files/etc/ssh/sshd_config/PermitRootLogin no
augtool> save
```

Augeas - in Puppet: Using Puppet with Augeas

augeas { "sshd_config":
 changes => [
 "set /files/etc/ssh/sshd_config/PermitRootLogin no",
 ],
}

cfengine: Force running shortly after a recent execution
```
cfagent -K
```
cfengine - Design Center: Git repository with sketches and examples for cfengine.
cfengine - cf-sketch: Find and install sketches from the Design Center repository
detox: Tool for recursive cleanup of file names.
```
detox -v -r <directory>
```
Chef - List Nodes per Role:
```
knife search node 'roles:<role name>'
```
Chef - Fix RabbitMQ 100% CPU usage
Chef - Edit Files: using a Script resource.
Chef - Manage Amazon EC2 instances
Chef - Tutorial on how to Setup Nagios in EC2

puppet: Debugging deployment and rules on a local machine. This only makes sense in "one time" mode running in one of the following variants:

puppetd --test # enable standard debugging options
puppetd --debug # enable full debugging
puppetd --one-time --detailed-exitcodes # Enable exit codes:
           # 2=changes applied
           # 4=failure

Database

Debian

Build Kernel Package: How to build kernel packages with make-pkg

cd /usr/src/linux && make-kpkg clean && make-kpkg --initrd --revision=myrev kernel_image

Setup Keyring: How to solve "The following packages cannot be authenticated"
```
 apt-get install debian-archive-keyring
apt-get update
```
Force remove broken "reportbug": This can happen during dist-upgrades from Etch/Sarge to Lenny.
Packages - Reconfigure after installation:
```
dpkg-reconfigure -a
```

dpkg Cheat-Sheet: Query package infos

# Resolve file to package
dpkg -S /etc/fstab

# Print all files of a package
dpkg -L passwd # provided files
dpkg -c passwd # owned files

# Find packages by name
dpkg -l gnome*

# Package details
dpkg -p passwd

Ubuntu - Access Repositories for older releases. Once a release is deprecated it is moved to old-releases.ubuntu.com. You need to adapt /etc/apt/sources.list to fetch packages from there
```
sed -i 's/archive.ubuntu.com/old-releases.ubuntu.com/' /etc/apt/sources.list
```

Debugging / Performance Tools

Drop Filesystem Cache
```
echo 1 > /proc/sys/vm/drop_caches
```

echo 1 > /proc/sys/vm/block_dump

# wait some time...
echo 0 > /proc/sys/vm/block_dump

# Now check syslog for block dump lines

dmesg -T      # Enable human readable timestamps
dmesg -x      # Show facility and log level
dmesg -f daemon     # Filter for facility daemon
dmesg -l err,crit,alert,emerg # Filter for errors

lslk - Find file locks: Use lslk to find which PID is blocking an flock() to a file.

lsof - Find owners of open file handles:

lsof      # Complete list
lsof -i :22    # Filter single TCP port
lsof [email protected]:22 # Filter single connection endpoint
lsof -u <user>   # Filter per user
lsof -c <name>   # Filter per process name
lsof -p 12345    # Filter by PID
lsof /etc/hosts   # Filter single file

Perf Tutorial: 2.6+ generic kernel performance statistics tool.
```
perf stat -B some_command
```
dstat: Replaces vmstat, iostat, netstat and ifstat and allows to determine PID that is most CPU and most I/O expensive
```
dstat -a --top-bio --top-cpu
```
iotop: Python script to monitor I/O like top
PHP - How to setup the APD debugger

Filesystem / Partitioning

uNetBootin: Create bootable media for any distribution. Most useful with USB sticks.
Convert ext2 to ext3:
```
tune2fs -j /dev/hda1
```

Convert ext3 to ext4:

tune2fs -O extents,uninit_bg,dir_index /dev/sda1

Determine Inode Count:
```
tune2fs -l /dev/sda1 | grep Inode
```
Disable ext4 barriers: Add "barrier=0" to the mount options.

LVM - Add another disk: How to add a disk to an existing volume

# Setup partition with (use parted for >2TB)
(parted) mklabel gpt       # only when >2TB
(parted) mkpart primary lvm 0 4T    # setup disk full size (e.g. 4TB)

pvcreate /dev/sdb1       # Create physical LVM disk
vgextend vg01 /dev/sdb1      # Add to volume group
vgextend -L +4t /dev/mapper/vg01-lvdata  # Extend your volume 
resize2fs /dev/mapper/vg01-lvdata   # Auto-resize file system

rsync - --delete doesn't work: It happens when you call rsync without a trailing slash in the source path like this:
```
rsync -az -e ssh --delete /data server:/data
```
It just won't delete anything. It will when running it like this:
```
rsync -az -e ssh --delete /data/ server:/data
```

Mail

postsuper -d ALL
postsuper -d ALL deferred

Middleware

Heartbeat - Manual IP Failover

# Either run on the node that should take over
/usr/share/heartbeat/hb_failover

# Or run on the node to should stop working
/usr/share/heartbeat/hb_standby

Pacemaker - Setup Steps

RabbitMQ - Commands

rabbitmqctl list_vhosts   # List all defined vhosts
rabbitmqctl list_queues <vhost> # List all queues for the vhost

rabbitmqctl report    # Dump detailed report on RabbitMQ instance

RabbitMQ - Fix Chef 100% CPU usage
RabbitMQ - Setup Clustering

Monitoring

Munin - Test Plugins:

/usr/sbin/munin-run <plugin name> # for values
/usr/sbin/munin-run <plugin name> config # for configuration

Munin - Manual Update Run:
```
sudo -u munin /usr/bin/munin-cron
```

date: Calculate Last Day of Month:

/usr/sbin/munin-node-configure --suggest

# and enable them using
/usr/sbin/munin-node-configure --shell | sh

Network

ethtool - Usage

ethtool eth0                       # Print general info on eth0
ethtool -i eth0                    # Print kernel module info
ethtool -S eth0                    # Print eth0 traffic statistics
ethtool -a eth0                    # Print RX, TX and auto-negotiation settings

# Changing NIC settings...
ethtool -s eth0 speed 100
ethtool -s eth0 autoneg off
ethtool -s eth0 duplex full
ethtool -s eth0 wol g               # Turn on wake-on-LAN

Do not forget to make changes permanent in e.g. /etc/network/interfaces.

NFS - Tuning Secrets: SGI Slides on NFS Performance

nttcp - TCP performance testing

# On sending host
nttcp -t -s

# On receiving host
nttcp -r -s

tcpdump - Be verbose and print full package hex dumps:

 tcpdump -i eth0 -nN -vvv -xX -s 1500 port <some port>

SNMP - Dump all MIBs: When you need to find the MIB for an object known only by name try
```
snmpwalk -c public -v 1 -O s <myhost> .iso | grep <search string>
```
Hurricane Electric - BGP Tools: Statistics on all AS as well as links to their looking glasses.

Package Management

Debian

apt-get install <package> 
apt-get remove <package> # Remove files installed by <package>
apt-get purge <package>  # Remove <package> and all the files it did create

apt-get upgrade    # Upgrade all packages
apt-get install <package> # Upgrade an install package

apt-get dist-upgrade  # Upgrade distribution

apt-cache search <package> # Check if there is such a package name in the repos
apt-cache clean    # Remove all downloaded .debs

dpkg -l      # List all installed/known packages

# More dpkg invocations above in the "Debian" section!

Ubuntu (like Debian) with the addition of

do-release-upgrade   # For Ubuntu release upgrades

OpenSuSE

zypper install <package> 

zypper refresh    # Update repository infos

zypper list-updates
zypper repos    # List configured repositories

zypper dist-upgrade   # Upgrade distribution
zypper dup     # Upgrade distribution (alias)

zypper search <package>  # Search for <package>
zypper search --search-descriptions <package>

zypper clean      # Clean package cache

# For safe updates:
zypper mr –keep-packages –remote # Enable caching of packages
zypper dup -D      # Fetch packages using a dry run
zypper mr –all –no-refresh  # Set cache usage for following dup
zypper dup      # Upgrade!

Redhat:
```
up2date
```

Centos:

yum update     # Upgrade distro
yum install <package>  # Install <package>

RAID

mdadm - Commands

cat /proc/mdstat   # Print status

mdadm --detail /dev/md0  # Print status per md

mdadm --manage -r /dev/md0 /dev/sda1 # Remove a disk
mdadm --zero-superblock /dev/sda1  # Initialize a disk
mdadm --manage -a /dev/md0 /dev/sda1 # Add a disk

mdadm --manage --set-faulty /dev/md0 /dev/sda1

hpacucli - Commands

# Show status of all arrays on all controllers
hpacucli all show config
hpacucli all show config detail

# Show status of specific controller
hpacucli ctrl=0 pd all show

# Show Smart Array status
hpacucli all show status

LSI MegaRAID - Commands

# Get number of controllers
/opt/MegaRAID/MegaCli/MegaCli64 -adpCount -NoLog

# Get number of logical drives on controller #0
/opt/MegaRAID/MegaCli/MegaCli64 -LdGetNum -a0 -NoLog

# Get info on logical drive #0 on controller #0
/opt/MegaRAID/MegaCli/MegaCli64 -LdInfo -L0 -a0 -NoLog

Security

chroot Best Practices
Debian - Package Vulnerability Scanner:
```
debsecan --suite=sid
```
FreeBSD - Package Vulnerability Scanner:
```
portaudit -Fda
```
Overview of Automated Linux Package Vulnerability Scanning
Debug Apparmor: How to determine which permissions prevent some type of access.
OpenVAS - Security Scanner (commercial)
TCP/IP Stack Hardening: Unix network hardening
OpenSSL Kurzreferenz: All commands to create keys, certificates and certificate requests.
Print x509 Certificate Infos:
```
openssl x509 -text -in my.crt
```
By replacing "x509" with "ca" or "crt" you can dump other file types too.
mkpasswd - Create Passwords

Shell

date: Convert To Unix Timestamp:
```
date -d "$date" +%s
```
date: Convert From Unix Timestamp:
```
date -d "1970-01-01 1234567890 sec GMT"
```

cal $(date "+%M %y") | grep -v ^$ | tail -1 | sed 's/^.* \([0-9]*\)$/\1/'

bash: Extend Completion: How to setup your own bash completion schemas.

complete -W 'add branch checkout clone commit diff grep init log merge mv pull push rebase rm show status tag' git

bash - Pass file descriptor insteaf of commands: This can be used with all tools that demand a file name paramter:
```
diff <(echo abc;echo def) <(echo abc;echo abc)
```

bash - Regexp matching:

if [[ "$string" =~ ^[0-9]+$ ]]; then 
    echo "Is a number"
fi

bash - Regexp match extraction variant #1: Note how you need to set the regexp into a variable because you must not quote it in the if condition!
```
REGEXP="2013:06:23 ([0-9]+):([0-9]+)"
if [[ "$string" =~ $REGEXP ]]; then
    echo "Hour ${BASH_REMATCH[1]} Minute ${BASH_REMATCH[2]}"
fi
```
bash - Regexp match extraction variant #2: Actually using "expr" can much simpler especially when only on value is to be extracted:
```
hour=$(expr match "$string" '2013:06:23 $[0-9]\+$')
```
bash - kill all childs on exit:
```
trap true TERM
kill -- -$$
```

bash - Control History Handling:

unset HISTFILE      # Stop logging history in this bash instance
HISTIGNORE="[ ]*"      # Do not log commands with leading spaces
HISTIGNORE="&"      # Do not log a command multiple times

HISTTIMEFORMAT="%Y-%m-%d %H:%M:%S" # Log with timestamps

bash - apply /etc/security/limits.conf change immediately:
```
sudo -i -u <user>
```
Mail Attachments: Dozens of variants to mail attachments using Unix tools.
tail -f until removed: When you want to tail a file until it gets removed
```
tail --follow=name myfile
```

join - DB-like joining of CSV files:

join -o1.2,2.3 -t ";" -1 1 -2 2 employee.csv tasks.csv

shell - list all commands:
```
compgen -c |sort -u
```
shell - Check for interactive terminal: Run "tty" in silent mode and check the exit code
```
tty -s
```
shell - ANSI color matrix
Sorting column: Use the -k switch of "sort" to sort lines by a column. E.g.
```
cat access.log | sort -k 1
```
watch: wait for file/directory changes and run a command
```
watch -d ls -l
```

Shell - Unbuffer Output:

stdbuf -i0 -o0 -e0 <some command>  # Best solution

unbuffer <some command>     # Wrapper script from expect

dos2unix with vi:
```
:%s/^V^M//g
```

SSH

authorized_keys HowTo: Syntax and options...
Easy Key Copying: Stop editing authorized_keys remote. Use the standard OpenSSH ssh-copy-id instead.
```
ssh-copy-id [-i keyfile] user@maschine
```
ProxyCommand: Run SSH over a gateway and forward to other hosts based and/or perform some type of authentication. In .ssh/config you can have:
```
Host unreachable_host
  ProxyCommand ssh gateway_host exec nc %h %p
```
Transparent Multi-Hop:
```
ssh host1 -A -t host2 -A -t host3 ...
```

100% non-interactive SSH: What parameters to use to avoid any interaction.

ssh -i my_priv_key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PreferredAuthentications=publickey user@host -n "/bin/ls"

SFTP chroot with umask: How to enforce a umask with SFTP
```
Subsystem sftp /usr/libexec/openssh/sftp-server -u 0002
```
Agent Forwarding explained with pictures! Configured in /etc/ssh_config with
```
Host *
ForwardAgent yes
```
How to use a SOCKS Proxy On the client start proxy by
```
ssh -D <port> <remote host>
```

Webserver Stack

Apache vs. Lighttpd Rewrite Rules: How to migrate.

Apache - Rewrite on File Pattern

RewriteCond %{REQUEST_FILENAME} (.*)\.(html|htm)$

Apache - Rewrite on User Agent

RewriteCond %{HTTP_USER_AGENT} (iPhone|iPad)

Apache - Exception Hook: Since 2.0.49 Apache has an exception hook to handle crashes.
```
EnableExceptionHook on
```
memcached - List of Monitoring Solutions
memcached - List of Telnet Commands

stats cachedump <slab class> <number of keys to dump>

memcached - Key Dump Tool "memdump"
memcached - Persistent Hashing: How to survive server failures without the whole cluster degrading. Of course applies to any cache.
memcachedb - Log Archiving: To clean unused memcachedb logs open a telnet session and run
```
db_archive
```
to clean unused log files

nginx - Complex Conditions: As nginx does not support complex logic in if() conditions you need to set flags in a smart way to workaround it.

# Define a control flag
set $extra_handling = 0;

# Set the control flag when needed
if ($variable1 ~* pattern) {
    set $extra_handling = 1;
}

# Unset the flag if needed
if ( $variable2 = 1 ) {
    set $extra_handling = 0;
}

if ( $extra_handling = 1 ) {
    # Trigger intended behaviour
}

Redis - Most Important Commands for Sysadmins

This is a list of non-trivial Linux administration commands and can be used as a cheat sheet or link collection. If you find errors or want to add something please post a comment below!

Automation Products

Which automation tools are actually out there?

Bcfg2: Alternative to puppet and cfengine by Argonne National Laboratory. (IMO out-dated)
cdist: configuration with shell scripting
cfengine (active, commercially backed, large user base)
Chef: Alternative to puppet (Ruby, active, commercially backed, large user base)
EMC UIM - Unified Infrastructure Manager, VCE VBlock (enterprise, commercial)
Puppet (Ruby-like + Ruby, active, commercially backed, large user base)
slaughter (Perl, active, small user base)
Sprinkle (Ruby, quite recent)
Rundeck - Workflow manager for node - role systems like EC2, chef, puppet ...
SaltStack - (Python, semi-commercial, new, small user base)

Finally it is worth to check the Wikipedia Comparison Chart for other less known and new tools!

Automation

Augeas: Very flexible file editor to be used with Puppet or standalone. Could also work with cfengine.
```
$ augtool
augtool> set /files/etc/ssh/sshd_config/PermitRootLogin no
augtool> save
```

Augeas - in Puppet: Using Puppet with Augeas

augeas { "sshd_config":
 changes => [
 "set /files/etc/ssh/sshd_config/PermitRootLogin no",
 ],
}

cfengine: Force running shortly after a recent execution
```
cfagent -K
```
cfengine - Design Center: Git repository with sketches and examples for cfengine.
cfengine - cf-sketch: Find and install sketches from the Design Center repository
detox: Tool for recursive cleanup of file names.
```
detox -v -r <directory>
```
Chef - Dry Run:
```
chef-client -Fmin --why-run
```
Chef - List System Info:
```
ohai
```
Chef - List Node Info:
```
knife node show <node>
```
Chef - List Nodes per Role:
```
knife search node 'roles:<role name>'
```
Chef - Fix RabbitMQ 100% CPU usage

Chef - knife + SSH:

knife ssh -a ipaddress name:server1 "chef-client"

you can also use patterns:

knife ssh -a ipaddress name:www* "uptime"

Chef - Edit Files: using a Script resource.
Chef - Manage Amazon EC2 instances
Chef - Tutorial on how to Setup Nagios in EC2

puppet: Debugging deployment and rules on a local machine. This only makes sense in "one time" mode running in one of the following variants:

puppetd --test # enable standard debugging options
puppetd --debug # enable full debugging
puppetd --one-time --detailed-exitcodes # Enable exit codes:
           # 2=changes applied
           # 4=failure

Software Firewalls, LBs

pfsense OpenBSD firewall
vyatta Linux firewall
Zen Load Balancer (virtual appliance)
HAProxy TCP LB

Install Servers

Cobbler
MAAS - Ubuntu "Metal As A Service" install server

Orchestration Tools

JuJu: mostly for Ubuntu, service orchestration tool (Python, commercially backed)
Maestro (enterprise, commercial)
mcollective - Puppet parallelizing and orchestration framework
SaltStack

Database

Debian

Build Kernel Package: How to build kernel packages with make-pkg

cd /usr/src/linux && make-kpkg clean && make-kpkg --initrd --revision=myrev kernel_image

Setup Keyring: How to solve "The following packages cannot be authenticated"
```
apt-get install debian-archive-keyring
apt-get update
```
Force remove broken "reportbug": This can happen during dist-upgrades from Etch/Sarge to Lenny.
Packages - Reconfigure after installation:
```
dpkg-reconfigure -a
```

dpkg Cheat-Sheet: Query package infos

# Resolve file to package
dpkg -S /etc/fstab

# Print all files of a package
dpkg -L passwd # provided files
dpkg -c passwd # owned files

# Find packages by name
dpkg -l gnome*

# Package details
dpkg -p passwd

Ubuntu - Access Repositories for older releases. Once a release is deprecated it is moved to old-releases.ubuntu.com. You need to adapt /etc/apt/sources.list to fetch packages from there
```
sed -i 's/archive.ubuntu.com/old-releases.ubuntu.com/' /etc/apt/sources.list
```

Debugging / Performance Tools

Reboot when /sbin is unusable
```
echo b >/proc/sysrq-trigger
```
List Context Switches per Process
```
pidstat -w
```
Drop Filesystem Cache
```
echo 1 > /proc/sys/vm/drop_caches
```

echo 1 > /proc/sys/vm/block_dump

# wait some time...
echo 0 > /proc/sys/vm/block_dump

# Now check syslog for block dump lines

Check for changed sysctl() settings:
```
sysctl -p
```

dmesg -T      # Enable human readable timestamps
dmesg -x      # Show facility and log level
dmesg -f daemon     # Filter for facility daemon
dmesg -l err,crit,alert,emerg # Filter for errors

lslk - Find file locks: Use lslk to find which PID is blocking an flock() to a file.

lsof - Find owners of open file handles:

lsof      # Complete list
lsof -i :22    # Filter single TCP port
lsof [email protected]:22 # Filter single connection endpoint
lsof -u <user>   # Filter per user
lsof -c <name>   # Filter per process name
lsof -p 12345    # Filter by PID
lsof /etc/hosts   # Filter single file

Perf Tutorial: 2.6+ generic kernel performance statistics tool.
```
perf stat -B some_command
```
dstat: Replaces vmstat, iostat, netstat and ifstat and allows to determine PID that is most CPU and most I/O expensive
```
dstat -a --top-bio --top-cpu
```
iotop: Python script to monitor I/O like top
PHP - How to setup the APD debugger

Filesystem / Partitioning

uNetBootin: Create bootable media for any distribution. Most useful with USB sticks.
Convert ext2 to ext3:
```
tune2fs -j /dev/hda1
```

Convert ext3 to ext4:

tune2fs -O extents,uninit_bg,dir_index /dev/sda1

Determine Inode Count:
```
tune2fs -l /dev/sda1 | grep Inode
```
Disable ext4 barriers: Add "barrier=0" to the mount options.

LVM - Add another disk: How to add a disk to an existing volume

# Setup partition with (use parted for >2TB)
(parted) mklabel gpt       # only when >2TB
(parted) mkpart primary lvm 0 4T    # setup disk full size (e.g. 4TB)

pvcreate /dev/sdb1       # Create physical LVM disk
vgextend vg01 /dev/sdb1      # Add to volume group
vgextend -L +4t /dev/mapper/vg01-lvdata  # Extend your volume 
resize2fs /dev/mapper/vg01-lvdata   # Auto-resize file system

rsync - --delete doesn't work: It happens when you call rsync without a trailing slash in the source path like this:
```
rsync -az -e ssh --delete /data server:/data
```
It just won't delete anything. It will when running it like this:
```
rsync -az -e ssh --delete /data/ server:/data
```

Hosting

Hoster Lookup: whoishosthingthis.com, who-hosts.com
iplist.net: Simple reverse lookup of neighbour IPs
Hoster Status: Status Channels for different hosters:
- Rackspace: Follow @RackStatus
- CloudFlare: Follow @CloudFlareSys
- Hetzner: Follow @hetznerstatus

Hardware Info

HP - Find Installed Memory:

dmidecode 2>&1 |grep -A17 -i "Memory Device" |egrep "Memory Device|Locator: PROC|Size" |grep -v "No Module Installed" |grep -A1 -B1 "Size:"

Mail

postsuper -d ALL
postsuper -d ALL deferred

Microsoft SPF Record Tool: SPF Record wizard

Middleware

Heartbeat - Manual IP Failover

# Either run on the node that should take over
/usr/share/heartbeat/hb_failover

# Or run on the node to should stop working
/usr/share/heartbeat/hb_standby

keepalived: Simple VRRP solution

Pacemaker - Commands

# Cluster Resource Status
crm_mon
crm_mon -1
crm_mon -f   # failure count

# Dump and Import Config
cibadmin --query --obj_type resources >file.xml
cibadmin --replace --obj_type resources --xml-file file.xml

# Resource Handling
crm resource stop <name>
crm resource start <name>
crm resource move <name> <node>

# Put entire cluster in maintenance
crm configure property maintenance-mode=true
crm configure property maintenance-mode=false

# Unmanaged Mode for single services
crm resource unmanage <name>
crm resource manage <name>

Pacemaker - Setup Steps

RabbitMQ - Commands

rabbitmqctl list_vhosts   # List all defined vhosts
rabbitmqctl list_queues <vhost> # List all queues for the vhost

rabbitmqctl report    # Dump detailed report on RabbitMQ instance  

# Plugin management
/usr/lib/rabbitmq/bin/rabbitmq-plugins enable <name>
/usr/lib/rabbitmq/bin/rabbitmq-plugins list

RabbitMQ - Fix Chef 100% CPU usage
RabbitMQ - Setup Clustering

wackamole - Commands

wackatrl -l     # List status
wackatrl -f     # Remove node from cluster
wackatrl -s     # Add node to cluster again

Monitoring

SEC - Simple Event Correlation

Munin - Test Plugins:

/usr/sbin/munin-run <plugin name> # for values
/usr/sbin/munin-run <plugin name> config # for configuration

/usr/sbin/munin-node-configure --suggest

# and enable them using
/usr/sbin/munin-node-configure --shell | sh

Munin Server - Manual Update Run:
```
sudo -u munin /usr/bin/munin-cron
```
SmokePing - Monitoring network latency and loss
dmesg Nagios Plugin

Network Administration Commands

Package Management

Debian

apt-get install <package> 
apt-get remove <package> # Remove files installed by <package>
apt-get purge <package>  # Remove <package> and all the files it did create

apt-get upgrade    # Upgrade all packages
apt-get install <package> # Upgrade an install package

apt-get dist-upgrade  # Upgrade distribution

apt-cache search <package> # Check if there is such a package name in the repos
apt-cache clean    # Remove all downloaded .debs

dpkg -l      # List all installed/known packages

# More dpkg invocations above in the "Debian" section!

Ubuntu (like Debian) with the addition of

# 1. Edit settings in  /etc/update-manager/release-upgrades
# e.g. set "Prompt=lts"

# 2. Run upgrade
do-release-upgrade -d   # For Ubuntu release upgrades

OpenSuSE

zypper install <package> 

zypper refresh    # Update repository infos

zypper list-updates
zypper repos    # List configured repositories

zypper dist-upgrade   # Upgrade distribution
zypper dup     # Upgrade distribution (alias)

zypper search <package>  # Search for <package>
zypper search --search-descriptions <package>

zypper clean      # Clean package cache

# For safe updates:
zypper mr –keep-packages –remote # Enable caching of packages
zypper dup -D      # Fetch packages using a dry run
zypper mr –all –no-refresh  # Set cache usage for following dup
zypper dup      # Upgrade!

Redhat:
```
up2date
```

Centos:

yum update     # Upgrade distro
yum install <package>  # Install <package>

RAID

mdadm - Commands

cat /proc/mdstat   # Print status

mdadm --detail /dev/md0  # Print status per md

mdadm --manage -r /dev/md0 /dev/sda1 # Remove a disk
mdadm --zero-superblock /dev/sda1  # Initialize a disk
mdadm --manage -a /dev/md0 /dev/sda1 # Add a disk

mdadm --manage --set-faulty /dev/md0 /dev/sda1

hpacucli - Commands

# Show status of all arrays on all controllers
hpacucli all show config
hpacucli all show config detail

# Show status of specific controller
hpacucli ctrl=0 pd all show

# Show Smart Array status
hpacucli all show status

LSI MegaRAID - Commands

# Get number of controllers
/opt/MegaRAID/MegaCli/MegaCli64 -adpCount -NoLog

# Get number of logical drives on controller #0
/opt/MegaRAID/MegaCli/MegaCli64 -LdGetNum -a0 -NoLog

# Get info on logical drive #0 on controller #0
/opt/MegaRAID/MegaCli/MegaCli64 -LdInfo -L0 -a0 -NoLog

Security

chroot Best Practices
Debian - Package Vulnerability Scanner:
```
debsecan --suite=sid
```
FreeBSD - Package Vulnerability Scanner:
```
portaudit -Fda
```
Overview of Automated Linux Package Vulnerability Scanning
Debug Apparmor: How to determine which permissions prevent some type of access.
OpenVAS - Security Scanner (commercial)
TCP/IP Stack Hardening: Unix network hardening
OpenSSL Kurzreferenz: All commands to create keys, certificates and certificate requests.
Print x509 Certificate Infos:
```
openssl x509 -text -in my.crt
```
By replacing "x509" with "ca" or "crt" you can dump other file types too.
mkpasswd - Create Passwords

Shell Scripting - Cheat Sheet

SSH

SSH Escape Key: Pressing "~?" (directly following a newline) gives a menu for escape sequences:

Supported escape sequences:
  ~.  - terminate connection (and any multiplexed sessions)
  ~B  - send a BREAK to the remote system
  ~C  - open a command line
  ~R  - Request rekey (SSH protocol 2 only)
  ~^Z - suspend ssh
  ~#  - list forwarded connections
  ~&  - background ssh (when waiting for connections to terminate)
  ~?  - this message
  ~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

SSH Mounting remote filesystem:

# To mount a remote home dir 
sshfs user@server: /mnt/home/user/

# Unmount again with
fuserumount -u /mnt/home/user

authorized_keys HowTo: Syntax and options...

Automatic Jump Host Proxying: Use the following ~/.ssh/config snippet and create ~/.ssh/tmp before using it

ControlMaster auto
ControlPath /home/<user name>/.ssh/tmp/%h_%p_%r
 
Host <your jump host>
  ForwardAgent yes
  Hostname <your jump host>
  User <your user name on jump host>

# Note the server list can have wild cards, e.g. "webserver-* database*"
Host <server list>
  ForwardAgent yes
  User <your user name on all these hosts>
  ProxyCommand ssh -q <your jump host> nc -q0 %h 22

Easy Key Copying: Stop editing authorized_keys remote. Use the standard OpenSSH ssh-copy-id instead.
```
ssh-copy-id [-i keyfile] user@maschine
```
ProxyCommand: Run SSH over a gateway and forward to other hosts based and/or perform some type of authentication. In .ssh/config you can have:
```
Host unreachable_host
  ProxyCommand ssh gateway_host exec nc %h %p
```
Transparent Multi-Hop:
```
ssh host1 -A -t host2 -A -t host3 ...
```

100% non-interactive SSH: What parameters to use to avoid any interaction.

ssh -i my_priv_key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PreferredAuthentications=publickey user@host -n "/bin/ls"

SFTP chroot with umask: How to enforce a umask with SFTP
```
Subsystem sftp /usr/libexec/openssh/sftp-server -u 0002
```
Agent Forwarding explained with pictures! Configured in /etc/ssh_config with
```
Host *
ForwardAgent yes
```
How to use a SOCKS Proxy On the client start proxy by
```
ssh -D <port> <remote host>
```

Parallel SSH on Debian

apt-get install pssh

and use it like this

pssh -h host_list.txt <args>

Clustered SSH on Debian

apt-get install clusterssh

and use it like this

cssh server1 server2

Webserver Stack

Alexa Pagerank API

http://data.alexa.com/data?cli=10&url=$DOMAIN

Apache vs. Lighttpd Rewrite Rules: How to migrate.

Apache - Rewrite on File Pattern

RewriteCond %{REQUEST_FILENAME} (.*)\.(html|htm)$

Apache - Rewrite on User Agent

RewriteCond %{HTTP_USER_AGENT} (iPhone|iPad)

Apache - Exception Hook: Since 2.0.49 Apache has an exception hook to handle crashes.
```
EnableExceptionHook on
```

Apache - Pipe CustomLog to a script:

 LoadModule logio_module modules/mod_logio.so

<IfModule mod_logio.c>
        CustomLog "| some-script.sh"  "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""
</IfModule>

DNS Prefetching HTML:

# Turning it on/off globally
<meta http-equiv="x-dns-prefetch-control" content="off">

# Turning it on per-domain
<link rel="dns-prefetch" href="http://www.spreadfirefox.com/">

Django Best Practices
memcached - List of Monitoring Solutions
memcached - List of Telnet Commands

stats cachedump <slab class> <number of keys to dump>

memcached - Key Dump Tool "memdump"
memcached - Persistent Hashing: How to survive server failures without the whole cluster degrading. Of course applies to any cache.
memcachedb - Log Archiving: To clean unused memcachedb logs open a telnet session and run
```
db_archive
```
to clean unused log files

nginx - Complex Conditions: As nginx does not support complex logic in if() conditions you need to set flags in a smart way to workaround it.

# Define a control flag
set $extra_handling = 0;

# Set the control flag when needed
if ($variable1 ~* pattern) {
    set $extra_handling = 1;
}

# Unset the flag if needed
if ( $variable2 = 1 ) {
    set $extra_handling = 0;
}

if ( $extra_handling = 1 ) {
    # Trigger intended behaviour
}

Redis - Most Important Commands for Sysadmins
Squid - Quick Test Proxy Setup

SSL - nginx BEAST mitigation:

ssl_ciphers RC4:HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;

HTTPS Client Test
HTTPS Server Test
SSL Overclocking
SSL Performance Case Study Cloudflare
Configure Forward Secrecy: HowTo for Apache, nginx and OpenSSL

Prevent out of ports on high traffic webserver due to FIN_WAIT and other issues:

# Ensure to maximize available ports
cat /proc/sys/net/ipv4/ip_local_port_range
echo 1024 65535 >/proc/sys/net/ipv4/ip_local_port_range

and set sockets to reuse

# sysctl -p
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

Linux HTTP/TCP Tuning for initcwnd
```
ip route change default via <gateway> dev eth0 initcwnd 10
```
consider also to increase net.ipv4.tcp_wmem[1]
initcwnd settings used by different CDNs

Automation - Products

Framework	DSL	CM	CM Encryption	Orchestration
cfengine	Propietary	?	?	Enterprise Only
Puppet	Ruby	Hiera	Hiera Eyaml	mcollective
Chef	Ruby	Builtin	Builtin	Pushy (knife plugin + ZeroMQ)
Saltstack	Python	Builtin	Builtin	Builtin

Other tools

Bcfg2: Alternative to puppet and cfengine by Argonne National Laboratory. (IMO out-dated)
cdist: configuration with shell scripting
EMC UIM - Unified Infrastructure Manager, VCE VBlock (enterprise, commercial)
slaughter (Perl, active, small user base)
Sprinkle (Ruby, quite recent)
Rundeck - Workflow manager for node - role systems like EC2, chef, puppet ...
IBM Tivoli

Finally it is worth to check the Wikipedia Comparison Chart for other less known and new tools!

Automation

Augeas: Very flexible file editor to be used with Puppet or standalone. Could also work with cfengine.
```
$ augtool
augtool> set /files/etc/ssh/sshd_config/PermitRootLogin no
augtool> save
```

Augeas - in Puppet: Using Puppet with Augeas

augeas { "sshd_config":
 changes => [
 "set /files/etc/ssh/sshd_config/PermitRootLogin no",
 ],
}

cfengine: Force running shortly after a recent execution
```
cfagent -K
```
cfengine - Design Center: Git repository with sketches and examples for cfengine.
cfengine - cf-sketch: Find and install sketches from the Design Center repository

Automation - Chef

Chef - Dry Run:
```
chef-client -Fmin --why-run
```
Chef - List System Info:
```
ohai
```
Chef - Bootstrap client:
```
knife bootstrap <FQDN/IP>
```

Chef - Change Run List:

knife node run_list <add|remove> <node> <cookbook>::<recipe>

Chef - List Node Info:
```
knife node show <node>
```
Chef - List Nodes per Role:
```
knife search node 'roles:<role name>'
```
Chef - Fix RabbitMQ 100% CPU usage

Chef - knife + SSH:

knife ssh -a ipaddress name:server1 "chef-client"

you can also use patterns:

knife ssh -a ipaddress name:www* "uptime"

Chef - Edit Files: using a Script resource.
Chef - Manage Amazon EC2 instances
Chef - Tutorial on how to Setup Nagios in EC2

puppet: Debugging deployment and rules on a local machine. This only makes sense in "one time" mode running in one of the following variants:

puppetd --test # enable standard debugging options
puppetd --debug # enable full debugging
puppetd --one-time --detailed-exitcodes # Enable exit codes:
           # 2=changes applied
           # 4=failure

Automation - Puppet

Bootstrap client

puppet agent -t --server <puppet master> [<options>]

Managing Certificates (on master)

puppet cert list
puppet cert list --all
puppet cert sign <name>
puppet cert clean <name>   # removes cert

Managing Modules

puppet module list
puppet module install <name>
puppet module uninstall <name>
puppet module upgrade <name>
puppet module search <name>

Inspecting Resources/Types

puppet describe -l
puppet resource <type name>

# Querying Examples
puppet resource user john.smith
puppet resource service apache
puppet resource mount /data
puppet resource file /etc/motd
puppet resource package wget

Gepetto: Puppet IDE
puppet - Correctly using Roles and Profiles

eyaml usage

eyaml encrypt -f <filename>
eyaml encrypt -s <string>
eyaml encrypt -p      # Encrypt password, will prompt for it

eyaml decrypt -f <filename>
eyaml decrypt -s <string>

eyaml edit -f <filename>    # Decrypts, launches in editor and reencrypts

mcollective commands

mco ping
mco ping -W "/some match pattern/"
mco ping -S "<some select query>"

# List agents, queries, plugins...
mco plugin doc
mco plugin doc <name>

mco rpc service start service=httpd
mco rpc service stop service=httpd

mco facts <keyword>

mco inventory <node name>

# With shell plugin installed
mco shell run <command>
mco shell run --tail <command>

mco shell start <command>    # Returns an id
mco shell watch <id>
mco shell kill <id>
mco shell list

Software Firewalls, LBs

pfsense OpenBSD firewall
vyatta Linux firewall
Zen Load Balancer (virtual appliance)
HAProxy TCP LB

Install Servers

Cobbler
MAAS - Ubuntu "Metal As A Service" install server
Foreman (integrated with puppet)

Orchestration Tools

JuJu: mostly for Ubuntu, service orchestration tool (Python, commercially backed)
Maestro (enterprise, commercial)
mcollective - Puppet parallelizing and orchestration framework
SaltStack

Database

Debian

Check for security upgrades

# With apt-show-versions
apt-show-versions | grep "security upgradeable"

# With aptitude
aptitude search '?and(~U,~Asecurity)'

Build Kernel Package: How to build kernel packages with make-pkg

cd /usr/src/linux && make-kpkg clean && make-kpkg --initrd --revision=myrev kernel_image

Setup Keyring: How to solve "The following packages cannot be authenticated"
```
apt-get install debian-archive-keyring
apt-get update
```
Force remove broken "reportbug": This can happen during dist-upgrades from Etch/Sarge to Lenny.
Packages - Reconfigure after installation:
```
dpkg-reconfigure -a
```

dpkg Cheat-Sheet: Query package infos

# Resolve file to package
dpkg -S /etc/fstab

# Print all files of a package
dpkg -L passwd # provided files
dpkg -c passwd # owned files

# Find packages by name
dpkg -l gnome*

# Package details
dpkg -p passwd

Ubuntu - Access Repositories for older releases. Once a release is deprecated it is moved to old-releases.ubuntu.com. You need to adapt /etc/apt/sources.list to fetch packages from there
```
sed -i 's/archive.ubuntu.com/old-releases.ubuntu.com/' /etc/apt/sources.list
```

Ubuntu - List Security Updates

# Print summary
/usr/lib/update-notifier/apt-check --human-readable

# Print package names
/usr/lib/update-notifier/apt-check -p

Ubuntu - Upgrade Security Fixes Only

apt-get dist-upgrade -o Dir::Etc::SourceList=/etc/apt/sources.security.repos.only.list

Debugging / Performance Tools

Reboot when /sbin is unusable
```
echo b >/proc/sysrq-trigger
```
List Context Switches per Process
```
pidstat -w
```
Drop Filesystem Cache
```
echo 1 > /proc/sys/vm/drop_caches
```

echo 1 > /proc/sys/vm/block_dump

# wait some time...
echo 0 > /proc/sys/vm/block_dump

# Now check syslog for block dump lines

Check for changed sysctl() settings:
```
sysctl -p
```

dmesg -T      # Enable human readable timestamps
dmesg -x      # Show facility and log level
dmesg -f daemon     # Filter for facility daemon
dmesg -l err,crit,alert,emerg # Filter for errors

lslk - Find file locks: Use lslk to find which PID is blocking an flock() to a file.

lsof - Find owners of open file handles:

lsof      # Complete list
lsof -i :22    # Filter single TCP port
lsof [email protected]:22 # Filter single connection endpoint
lsof -u <user>   # Filter per user
lsof -c <name>   # Filter per process name
lsof -p 12345    # Filter by PID
lsof /etc/hosts   # Filter single file

Perf Tutorial: 2.6+ generic kernel performance statistics tool.
```
perf stat -B some_command
```
dstat: Replaces vmstat, iostat, netstat and ifstat and allows to determine PID that is most CPU and most I/O expensive
```
dstat -a --top-bio --top-cpu
```
iotop: Python script to monitor I/O like top
PHP - How to setup the APD debugger

PHP - How to build Debian package for modules from PECL

apt-get install dh-make-php
dh-make-pecl <module name>
cd <source directory>
debuild
# .deb package will be in ...

Sysdig: Some of the project examples

sysdig fd.name contains /etc
sysdig -c topscalls_time    # Top system calls
sysdig -c topfiles_time proc.name=httpd    # Top files by process
sysdig -c topfiles_bytes     # Top I/O per file
sysdig -c fdcount_by fd.cip "evt.type=accept"   # Top connections by IP
sysdig -c fdbytes_by fd.cip  # Top bytes per IP

# Sick MySQL check via Apache
sysdig -A -c echo_fds fd.sip=192.168.30.5 and proc.name=apache2 and evt.buffer contains SELECT

sysdig -cl # List plugins
sysdig -c bottlenecks  # Run bottlenecks plugin

Filesystem / Partitioning

detox: Tool for recursive cleanup of file names.
```
detox -v -r <directory>
```

Fast File Deletion:

perl -e 'for(<*>){((stat)[9]<(unlink))}'

POSIX ACLs:

getfacl <file>  # List ACLs for file 
setfacl -m user:joe:rwx dir # Modify ACL
ls -ld <file>    # Check for active ACL (indicates a "+")

uNetBootin: Create bootable media for any distribution. Most useful with USB sticks.
Convert ext2 to ext3:
```
tune2fs -j /dev/hda1
```

Convert ext3 to ext4:

tune2fs -O extents,uninit_bg,dir_index /dev/sda1

Determine Inode Count:
```
tune2fs -l /dev/sda1 | grep Inode
```
Disable ext4 barriers: Add "barrier=0" to the mount options.

LVM - Add another disk: How to add a disk to an existing volume

# Setup partition with (use parted for >2TB)
(parted) mklabel gpt       # only when >2TB
(parted) mkpart primary lvm 0 4T    # setup disk full size (e.g. 4TB)

pvcreate /dev/sdb1       # Create physical LVM disk
vgextend vg01 /dev/sdb1      # Add to volume group
vgextend -L +4t /dev/mapper/vg01-lvdata  # Extend your volume 
resize2fs /dev/mapper/vg01-lvdata   # Auto-resize file system

rsync - --delete doesn't work: It happens when you call rsync without a trailing slash in the source path like this:
```
rsync -az -e ssh --delete /data server:/data
```
It just won't delete anything. It will when running it like this:
```
rsync -az -e ssh --delete /data/ server:/data
```

Hosting

Hoster Lookup: whoishosthingthis.com, who-hosts.com
iplist.net: Simple reverse lookup of neighbour IPs
Hoster Status: Status Channels for different hosters:
- Rackspace: Follow @RackStatus
- CloudFlare: Follow @CloudFlareSys
- Hetzner: Follow @hetznerstatus

Hardware Info

HP - Find Installed Memory:

dmidecode 2>&1 |grep -A17 -i "Memory Device" |egrep "Memory Device|Locator: PROC|Size" |grep -v "No Module Installed" |grep -A1 -B1 "Size:"

Mail

postsuper -d ALL
postsuper -d ALL deferred

Postfix - OpenDKIM Setup on Ubuntu
Postfix - SPF Setup on Ubuntu
Microsoft SPF Record Tool: SPF Record wizard
mxtoolbox.com - RBL test an IP

Middleware

Heartbeat - Manual IP Failover

# Either run on the node that should take over
/usr/share/heartbeat/hb_failover

# Or run on the node to should stop working
/usr/share/heartbeat/hb_standby

keepalived: Simple VRRP solution

Pacemaker - Commands

# Cluster Resource Status
crm_mon
crm_mon -1
crm_mon -f   # failure count

# Dump and Import Config
cibadmin --query --obj_type resources >file.xml
cibadmin --replace --obj_type resources --xml-file file.xml

# Resource Handling
crm resource stop <name>
crm resource start <name>
crm resource move <name> <node>

# Put entire cluster in maintenance
crm configure property maintenance-mode=true
crm configure property maintenance-mode=false

# Unmanaged Mode for single services
crm resource unmanage <name>
crm resource manage <name>

Pacemaker - Setup Steps

RabbitMQ - Commands

rabbitmqctl list_vhosts   # List all defined vhosts
rabbitmqctl list_queues <vhost> # List all queues for the vhost

rabbitmqctl report    # Dump detailed report on RabbitMQ instance  

# Plugin management
/usr/lib/rabbitmq/bin/rabbitmq-plugins enable <name>
/usr/lib/rabbitmq/bin/rabbitmq-plugins list

RabbitMQ - Fix Chef 100% CPU usage
RabbitMQ - Setup Clustering

wackamole - Commands

wackatrl -l     # List status
wackatrl -f     # Remove node from cluster
wackatrl -s     # Add node to cluster again

Monitoring

SEC - Simple Event Correlation

Munin - Test Plugins:

/usr/sbin/munin-run <plugin name> # for values
/usr/sbin/munin-run <plugin name> config # for configuration

/usr/sbin/munin-node-configure --suggest

# and enable them using
/usr/sbin/munin-node-configure --shell | sh

Munin Server - Manual Update Run:
```
sudo -u munin /usr/bin/munin-cron
```
SmokePing - Monitoring network latency and loss
dmesg Nagios Plugin

Network Administration Commands

Package Management

Debian File Diversion:

# Register diverted path and move away
dpkg-divert --add --rename --divert <renamed file path> &file path>

# Remove a diversion again (remove file first!)
dpkg-divert --rename --remove <file path>

Debian

apt-get install <package> 
apt-get remove <package> # Remove files installed by <package>
apt-get purge <package>  # Remove <package> and all the files it did create

apt-get upgrade    # Upgrade all packages
apt-get install <package> # Upgrade an install package

apt-get dist-upgrade  # Upgrade distribution

apt-cache search <package> # Check if there is such a package name in the repos
apt-cache clean    # Remove all downloaded .debs

dpkg -l      # List all installed/known packages

# More dpkg invocations above in the "Debian" section!

Ubuntu (like Debian) with the addition of

# 1. Edit settings in  /etc/update-manager/release-upgrades
# e.g. set "Prompt=lts"

# 2. Run upgrade
do-release-upgrade -d   # For Ubuntu release upgrades

Ubuntu: Unattended Upgrades

apt-get install unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades 
# and maybe set notification mail address in /etc/apt/apt.conf.d/50unattended-upgrades

OpenSuSE

zypper install <package> 

zypper refresh    # Update repository infos

zypper list-updates
zypper repos    # List configured repositories

zypper dist-upgrade   # Upgrade distribution
zypper dup     # Upgrade distribution (alias)

zypper search <package>  # Search for <package>
zypper search --search-descriptions <package>

zypper clean      # Clean package cache

# For safe updates:
zypper mr –keep-packages –remote # Enable caching of packages
zypper dup -D      # Fetch packages using a dry run
zypper mr –all –no-refresh  # Set cache usage for following dup
zypper dup      # Upgrade!

Redhat:
```
up2date
```

Centos:

yum update     # Upgrade distro
yum install <package>  # Install <package>

RAID

mdadm - Commands

cat /proc/mdstat   # Print status

mdadm --detail /dev/md0  # Print status per md

mdadm --manage -r /dev/md0 /dev/sda1 # Remove a disk
mdadm --zero-superblock /dev/sda1  # Initialize a disk
mdadm --manage -a /dev/md0 /dev/sda1 # Add a disk

mdadm --manage --set-faulty /dev/md0 /dev/sda1

hpacucli - Commands

# Show status of all arrays on all controllers
hpacucli all show config
hpacucli all show config detail

# Show status of specific controller
hpacucli ctrl=0 pd all show

# Show Smart Array status
hpacucli all show status

# Create new Array
hpacucli ctrl slot=0 create type=logicaldrive drives=1I:1:3,1I:1:4 raid=1

LSI MegaRAID - Commands

# Get number of controllers
/opt/MegaRAID/MegaCli/MegaCli64 -adpCount -NoLog

# Get number of logical drives on controller #0
/opt/MegaRAID/MegaCli/MegaCli64 -LdGetNum -a0 -NoLog

# Get info on logical drive #0 on controller #0
/opt/MegaRAID/MegaCli/MegaCli64 -LdInfo -L0 -a0 -NoLog

Security

chroot Best Practices
Debian - Package Vulnerability Scanner:
```
debsecan --suite=sid
```
FreeBSD - Package Vulnerability Scanner:
```
portaudit -Fda
```
Overview of Automated Linux Package Vulnerability Scanning
Debug Apparmor: How to determine which permissions prevent some type of access.
OpenVAS - Security Scanner (commercial)
TCP/IP Stack Hardening: Unix network hardening
Tiger - Unix auditing tool
OpenSSL Kurzreferenz: All commands to create keys, certificates and certificate requests.
Print x509 Certificate Infos:
```
openssl x509 -text -in my.crt
```
By replacing "x509" with "ca" or "crt" you can dump other file types too.
mkpasswd - Create Passwords

Shell Scripting - Cheat Sheet

SSH

SSH Escape Key: Pressing "~?" (directly following a newline) gives a menu for escape sequences:

Supported escape sequences:
  ~.  - terminate connection (and any multiplexed sessions)
  ~B  - send a BREAK to the remote system
  ~C  - open a command line
  ~R  - Request rekey (SSH protocol 2 only)
  ~^Z - suspend ssh
  ~#  - list forwarded connections
  ~&  - background ssh (when waiting for connections to terminate)
  ~?  - this message
  ~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

SSH Mounting remote filesystem:

# To mount a remote home dir 
sshfs user@server: /mnt/home/user/

# Unmount again with
fuserumount -u /mnt/home/user

authorized_keys HowTo: Syntax and options...

Automatic Jump Host Proxying: Use the following ~/.ssh/config snippet and create ~/.ssh/tmp before using it

ControlMaster auto
ControlPath /home/<user name>/.ssh/tmp/%h_%p_%r
 
Host <your jump host>
  ForwardAgent yes
  Hostname <your jump host>
  User <your user name on jump host>

# Note the server list can have wild cards, e.g. "webserver-* database*"
Host <server list>
  ForwardAgent yes
  User <your user name on all these hosts>
  ProxyCommand ssh -q <your jump host> nc -q0 %h 22

Easy Key Copying: Stop editing authorized_keys remote. Use the standard OpenSSH ssh-copy-id instead.
```
ssh-copy-id [-i keyfile] user@maschine
```
ProxyCommand: Run SSH over a gateway and forward to other hosts based and/or perform some type of authentication. In .ssh/config you can have:
```
Host unreachable_host
  ProxyCommand ssh gateway_host exec nc %h %p
```
Transparent Multi-Hop:
```
ssh host1 -A -t host2 -A -t host3 ...
```

100% non-interactive SSH: What parameters to use to avoid any interaction.

ssh -i my_priv_key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PreferredAuthentications=publickey user@host -n "/bin/ls"

SFTP chroot with umask: How to enforce a umask with SFTP
```
Subsystem sftp /usr/libexec/openssh/sftp-server -u 0002
```
Agent Forwarding explained with pictures! Configured in /etc/ssh_config with
```
Host *
ForwardAgent yes
```
How to use a SOCKS Proxy On the client start proxy by
```
ssh -D <port> <remote host>
```

Parallel SSH on Debian

apt-get install pssh

and use it like this

pssh -h host_list.txt <args>

Clustered SSH on Debian

apt-get install clusterssh

and use it like this

cssh server1 server2

Vim Remote File Editing:

vim scp://user@host//some/directory/file.txt

Webserver Stack

Alexa Pagerank API

http://data.alexa.com/data?cli=10&url=$DOMAIN

Apache vs. Lighttpd Rewrite Rules: How to migrate.

Apache - Rewrite on File Pattern

RewriteCond %{REQUEST_FILENAME} (.*)\.(html|htm)$

Apache - Rewrite on User Agent

RewriteCond %{HTTP_USER_AGENT} (iPhone|iPad)

Apache - Exception Hook: Since 2.0.49 Apache has an exception hook to handle crashes.
```
EnableExceptionHook on
```

Apache - Pipe CustomLog to a script:

 LoadModule logio_module modules/mod_logio.so

<IfModule mod_logio.c>
        CustomLog "| some-script.sh"  "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""
</IfModule>

DNS Prefetching HTML:

# Turning it on/off globally
<meta http-equiv="x-dns-prefetch-control" content="off">

# Turning it on per-domain
<link rel="dns-prefetch" href="http://www.spreadfirefox.com/">

Apache Zookeeper CLI

# Connect with
zkCli.sh -server 127.0.0.1:2181

# Commands
ls <path>
get <path>
set <path> 
delete <path>

Django Best Practices

HTTPS - Perfect Forward Secrecy (PFS): Use

    TLS_ECDHE_RSA_WITH_RC4_128_SHA
    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
    TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA

memcached - List of Monitoring Solutions
memcached - List of Telnet Commands