Linux Sysadmin Links

This is a list of non-trivial Linux administration commands and can be used as a cheat sheet or link collection. If you find errors or want to add something please post a comment below!

Automation Products

Which automation tools are actually out there?

• Bcfg2: Alternative to puppet and cfengine by Argonne National Laboratory. (IMO out-dated)
• cfengine (active, commercially backed, large user base)
• Chef: Alternative to puppet (Ruby, active, commercially backed, large user base)
• JuJu: mostly for Ubuntu, service orchestration tool (Python, commercially backed)
• Puppet (Ruby-like + Ruby, active, commercially backed, large user base)
• slaughter (Perl, active, small user base)
• Sprinkle (Ruby, quite recent)
• Wikipedia Comparison Chart: Check here for other less known and new tools!

Administration Automatisation

• Augeas: Very flexible file editor to be used with Puppet or standalone. Could also work with cfengine. $ augtool
augtool> set /files/etc/ssh/sshd_config/PermitRootLogin no
augtool> save
• Augeas - in Puppet: Using Puppet with Augeas
  

  augeas { "sshd_config":
   changes => [
   "set /files/etc/ssh/sshd_config/PermitRootLogin no",
   ],
  }

• cfengine: Force running shortly after a recent execution cfagent -K
• cfengine - Design Center: Git repository with sketches and examples for cfengine.
• cfengine - cf-sketch: Find and install sketches from the Design Center repository
• detox: Tool for recursive cleanup of file names. detox -v -r <directory>
• Chef - List Nodes per Role:
  

  knife search node 'roles:<role name>'

• Chef - Fix RabbitMQ 100% CPU usage
• Chef - Edit Files: using a Script resource.
• Chef - Manage Amazon EC2 instances
• Chef - Tutorial on how to Setup Nagios in EC2
• puppet: Debugging deployment and rules on a local machine. This only makes sense in "one time" mode running in one of the following variants:
  

  puppetd --test # enable standard debugging options
  puppetd --debug # enable full debugging
  puppetd --one-time --detailed-exitcodes # Enable exit codes:
             # 2=changes applied
             # 4=failure
  

Database

• MySQL - Administration Commands
• TCP/IP tuning for scaling
• pgbouncer - Debug Pooler Error
• Postgres - Regular Expressions: You can edit column using regular expressions by running regexp_replace()
  

  UPDATE table SET field=regexp_replace(field, 'match pattern', 'replace string', 'g');

• Postgres - List Settings:
  

  SHOW ALL;

• Postgres - Analyze Queries:
  

  EXPLAIN ANALYZE <sql statement>;

• Postgres - Show Running Queries:
  

  SELECT * FROM pg_stat_activity;

• Postgres - Kill Query:
  

  # 1.) Find query and its PID
  SELECT procpid, current_query FROM pg_stat_activity;
  
  # 2.) Kill the PID on the Unix shell

• Postgres - Set Backup Mode To be able to copy Postgres files e.g. to a slave or a backup you need to put the server into backup mode.
  

  SELECT pg_start_backup('label', true);
  SELECT pg_stop_backup();

Debian

• Build Kernel Package: How to build kernel packages with make-pkg cd /usr/src/linux && make-kpkg clean && make-kpkg --initrd --revision=myrev kernel_image
• Setup Keyring: How to solve "The following packages cannot be authenticated" apt-get install debian-archive-keyring
apt-get update
• Force remove broken "reportbug": This can happen during dist-upgrades from Etch/Sarge to Lenny.
• Packages - Reconfigure after installation: dpkg-reconfigure -a
• dpkg Cheat-Sheet: Query package infos
  

  # Resolve file to package
  dpkg -S /etc/fstab
  
  # Print all files of a package
  dpkg -L passwd # provided files
  dpkg -c passwd # owned files
  
  # Find packages by name
  dpkg -l gnome*
  
  # Package details
  dpkg -p passwd
  

• Ubuntu - Access Repositories for older releases. Once a release is deprecated it is moved to old-releases.ubuntu.com. You need to adapt /etc/apt/sources.list to fetch packages from there
  

  sed -i 's/archive.ubuntu.com/old-releases.ubuntu.com/' /etc/apt/sources.list

Debugging / Performance Tools

• dmesg - Filtering Output:
  

  dmesg -T      # Enable human readable timestamps
  dmesg -x      # Show facility and log level
  dmesg -f daemon     # Filter for facility daemon
  dmesg -l err,crit,alert,emerg # Filter for errors
  

• lslk - Find file locks: Use lslk to find which PID is blocking an flock() to a file.
• lsof - Find owners of open file handles:
  

  lsof      # Complete list
  lsof -i :22    # Filter single TCP port
  lsof -i@192.168.1.5:22 # Filter single connection endpoint
  lsof -u <user>   # Filter per user
  lsof -c <name>   # Filter per process name
  lsof -p 12345    # Filter by PID
  lsof /etc/hosts   # Filter single file
  

• Perf Tutorial: 2.6+ generic kernel performance statistics tool. perf stat -B some_command
• dstat: Replaces vmstat, iostat, netstat and ifstat and allows to determine PID that is most CPU and most I/O expensive dstat -a --top-bio --top-cpu
• iotop: Python script to monitor I/O like top
• PHP - How to setup the APD debugger

Filesystem / Partitioning

• uNetBootin: Create bootable media for any distribution. Most useful with USB sticks.
• Convert ext2 to ext3: tune2fs -j /dev/hda1
• Convert ext3 to ext4: tune2fs -O extents,uninit_bg,dir_index /dev/sda1
• Determine Inode Count: tune2fs -l /dev/sda1 | grep Inode
• Disable ext4 barriers: Add "barrier=0" to the mount options.
• LVM - Add another disk: How to add a disk to an existing volume
  

  # Setup partition with (use parted for >2TB)
  (parted) mklabel gpt       # only when >2TB
  (parted) mkpart primary lvm 0 4T    # setup disk full size (e.g. 4TB)
  
  pvcreate /dev/sdb1       # Create physical LVM disk
  vgextend vg01 /dev/sdb1      # Add to volume group
  vgextend -L +4t /dev/mapper/vg01-lvdata  # Extend your volume 
  resize2fs /dev/mapper/vg01-lvdata   # Auto-resize file system

• rsync - --delete doesn't work: How to debug this.

Mail

• Postfix - Drop Mails
  

  postsuper -d ALL
  postsuper -d ALL deferred

Middleware

• RabbitMQ - Commands
  

  rabbitmqctl list_vhosts   # List all defined vhosts
  rabbitmqctl list_queues <vhost> # List all queues for the vhost
  
  rabbitmqctl report    # Dump detailed report on RabbitMQ instance  
  

• RabbitMQ - Fix Chef 100% CPU usage

Monitoring

• Munin - Test Plugins:
  

  /usr/sbin/munin-run <plugin name> # for values
  /usr/sbin/munin-run <plugin name> config # for configuration

• Munin - Manual Update Run:
  

  sudo -u munin /usr/bin/munin-cron

• Munin - Test available plugins
  

  /usr/sbin/munin-node-configure --suggest
  
  # and enable them using
  /usr/sbin/munin-node-configure --shell | sh

Network

• tcpdump - Be verbose and print full package hex dumps:
  

   tcpdump -i eth0 -nN -vvv -xX -s 1500 port <some port>

• SNMP - Dump all MIBs: When you need to find the MIB for an object known only by name try snmpwalk -c public -v 1 -O s <myhost> .iso | grep <search string>
• Hurricane Electric - BGP Tools: Statistics on all AS as well as links to their looking glasses.

Package Management

• Debian
  

  apt-get install <package> 
  apt-get remove <package> # Remove files installed by <package>
  apt-get purge <package>  # Remove <package> and all the files it did create
  
  apt-get upgrade    # Upgrade all packages
  apt-get install <package> # Upgrade an install package
  
  apt-get dist-upgrade  # Upgrade distribution
  
  apt-cache search <package> # Check if there is such a package name in the repos
  apt-cache clean    # Remove all downloaded .debs
  
  dpkg -l      # List all installed/known packages
  
  # More dpkg invocations above in the "Debian" section!
  

• Ubuntu (like Debian) with the addition of
  

  do-release-upgrade   # For Ubuntu release upgrades

• OpenSuSE
  

  zypper install <package> 
  
  zypper refresh    # Update repository infos
  
  zypper list-updates
  zypper repos    # List configured repositories
  
  zypper dist-upgrade   # Upgrade distribution
  zypper dup     # Upgrade distribution (alias)
  
  zypper search <package>  # Search for <package>
  zypper search --search-descriptions <package>
  
  zypper clean      # Clean package cache
  
  # For safe updates:
  zypper mr –keep-packages –remote # Enable caching of packages
  zypper dup -D      # Fetch packages using a dry run
  zypper mr –all –no-refresh  # Set cache usage for following dup
  zypper dup      # Upgrade!
  

• Redhat:
  

  up2date

• Centos:
  

  yum update     # Upgrade distro
  yum install <package>  # Install <package>

RAID

• mdadm - Commands
  

  cat /proc/mdstat   # Print status
  
  mdadm --detail /dev/md0  # Print status per md
  
  mdadm --manage -r /dev/md0 /dev/sda1 # Remove a disk
  mdadm --zero-superblock /dev/sda1  # Initialize a disk
  mdadm --manage -a /dev/md0 /dev/sda1 # Add a disk
  
  mdadm --manage --set-faulty /dev/md0 /dev/sda1
  

• hpacucli - Commands
  

  # Show status of all arrays on all controllers
  hpacucli all show config
  hpacucli all show config detail
  
  # Show status of specific controller
  hpacucli ctrl=0 pd all show
  
  # Show Smart Array status
  hpacucli all show status
  

• LSI MegaRAID - Commands
  

  # Get number of controllers
  /opt/MegaRAID/MegaCli/MegaCli64 -adpCount -NoLog
  
  # Get number of logical drives on controller #0
  /opt/MegaRAID/MegaCli/MegaCli64 -LdGetNum -a0 -NoLog
  
  # Get info on logical drive #0 on controller #0
  /opt/MegaRAID/MegaCli/MegaCli64 -LdInfo -L0 -a0 -NoLog
  

Security

• chroot Best Practices
• Debian - Package Vulnerability Scanner:
  

  debsecan --suite=sid

• FreeBSD - Package Vulnerability Scanner:
  

  portaudit -Fda

• Overview of Automated Linux Package Vulnerability Scanning
• Debug Apparmor: How to determine which permissions prevent some type of access.
• OpenVAS - Security Scanner (commercial)
• TCP/IP Stack Hardening: Unix network hardening
• OpenSSL Kurzreferenz: All commands to create keys, certificates and certificate requests.
• Print x509 Certificate Infos: openssl x509 -text -in my.crt By replacing "x509" with "ca" or "crt" you can dump other file types too.
• mkpasswd - Create Passwords

Shell

• date: Convert To Unix Timestamp: date -d "$date" +%s
• date: Convert From Unix Timestamp: date -d "1970-01-01 1234567890 sec GMT"
• date: Calculate Last Day of Month: cal $(date "+%M %y") | grep -v ^$ | tail -1 | sed 's/^.* \([0-9]*\)$/\1/'
• bash: Extend Completion: How to setup your own bash completion schemas. complete -W 'add branch checkout clone commit diff grep init log merge mv pull push rebase rm show status tag' git
• bash - kill all childs on exit: trap true TERM
kill -- -$$
• bash - Control History Handling:
  

  unset HISTFILE      # Stop logging history in this bash instance
  HISTIGNORE="[ ]*"      # Do not log commands with leading spaces
  HISTIGNORE="&"      # Do not log a command multiple times
  
  HISTTIMEFORMAT="%Y-%m-%d %H:%M:%S" # Log with timestamps
  

• bash - apply /etc/security/limits.conf change immediately: sudo -i -u <user>
• Mail Attachments: Dozens of variants to mail attachments using Unix tools.
• tail -f until removed: When you want to tail a file until it gets removed tail --follow=name myfile
• join - DB-like joining of CSV files:
  

  join -o1.2,2.3 -t ";" -1 1 -2 2 employee.csv tasks.csv

• shell - list all commands: compgen -c |sort -u
• shell - Check for interactive terminal: Run "tty" in silent mode and check the exit code tty -s
• shell - ANSI color matrix
• Sorting column: Use the -k switch of "sort" to sort lines by a column. E.g.
  

  cat access.log | sort -k 1

• watch: wait for file/directory changes and run a command watch -d ls -l
• Shell - Unbuffer Output:
  

  stdbuf -i0 -o0 -e0 <some command>  # Best solution
  
  unbuffer <some command>     # Wrapper script from expect
  

• dos2unix with vi:
  

  :%s/^V^M//g

SSH

• authorized_keys HowTo: Syntax and options...
• Easy Key Copying: Stop editing authorized_keys remote. Use the standard OpenSSH ssh-copy-id instead.
  

  ssh-copy-id [-i keyfile] user@maschine

• ProxyCommand: Run SSH over a gateway and forward to other hosts based and/or perform some type of authentication. In .ssh/config you can have:
  

  Host unreachable_host
    ProxyCommand ssh gateway_host exec nc %h %p

• Transparent Multi-Hop: ssh host1 -A -t host2 -A -t host3 ...
• 100% non-interactive SSH: What parameters to use to avoid any interaction. ssh -i my_priv_key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PreferredAuthentications=publickey user@host -n "/bin/ls"
• SFTP chroot with umask: How to enforce a umask with SFTP Subsystem sftp /usr/libexec/openssh/sftp-server -u 0002
• Agent Forwarding explained with pictures! Configured in /etc/ssh_config with
  

  Host *
  ForwardAgent yes

• How to use a SOCKS Proxy On the client start proxy by
  

  ssh -D <port> <remote host>

Webserver Stack

• Apache vs. Lighttpd Rewrite Rules: How to migrate.
• Apache - Exception Hook: Since 2.0.49 Apache has an exception hook to handle crashes.
  

  EnableExceptionHook on

• memcached - List of Monitoring Solutions
• memcached - List of Telnet Commands
• memcached - How to Dump Keys
  

  stats cachedump <slab class> <number of keys to dump>

• memcached - Key Dump Tool "memdump"
• memcached - Persistent Hashing: How to survive server failures without the whole cluster degrading. Of course applies to any cache.