Linux Sysadmin Links

This is a list of non-trivial Linux administration commands and can be used as a cheat sheet or link collection. If you find errors or want to add something please post a comment below!

Automation Products

Which automation tools are actually out there?

  • Bcfg2: Alternative to puppet and cfengine by Argonne National Laboratory. (IMO out-dated)
  • cdist: configuration with shell scripting
  • cfengine (active, commercially backed, large user base)
  • Chef: Alternative to puppet (Ruby, active, commercially backed, large user base)
  • EMC UIM - Unified Infrastructure Manager, VCE VBlock (enterprise, commercial)
  • Puppet (Ruby-like + Ruby, active, commercially backed, large user base)
  • slaughter (Perl, active, small user base)
  • Sprinkle (Ruby, quite recent)
  • Rundeck - Workflow manager for node - role systems like EC2, chef, puppet ...
  • SaltStack - (Python, semi-commercial, new, small user base)

Finally it is worth to check the Wikipedia Comparison Chart for other less known and new tools!

Automation

  • Augeas: Very flexible file editor to be used with Puppet or standalone. Could also work with cfengine.
    $ augtool
    augtool> set /files/etc/ssh/sshd_config/PermitRootLogin no
    augtool> save
  • Augeas - in Puppet: Using Puppet with Augeas
    augeas { "sshd_config":
     changes => [
     "set /files/etc/ssh/sshd_config/PermitRootLogin no",
     ],
    }
  • cfengine: Force running shortly after a recent execution
    cfagent -K
  • cfengine - Design Center: Git repository with sketches and examples for cfengine.
  • cfengine - cf-sketch: Find and install sketches from the Design Center repository
  • Chef - Dry Run:
    chef-client -Fmin --why-run
  • Chef - List System Info:
    ohai
  • Chef - Change Run List:
    knife node run_list <add|remove> <node> <cookbook>::<recipe>
  • Chef - List Node Info:
    knife node show <node>
  • Chef - List Nodes per Role:
    knife search node 'roles:<role name>'
  • Chef - Fix RabbitMQ 100% CPU usage
  • Chef - knife + SSH:
    knife ssh -a ipaddress name:server1 "chef-client"

    you can also use patterns:

    knife ssh -a ipaddress name:www* "uptime"
  • Chef - Edit Files: using a Script resource.
  • Chef - Manage Amazon EC2 instances
  • Chef - Tutorial on how to Setup Nagios in EC2
  • puppet: Debugging deployment and rules on a local machine. This only makes sense in "one time" mode running in one of the following variants:
    puppetd --test # enable standard debugging options
    puppetd --debug # enable full debugging
    puppetd --one-time --detailed-exitcodes # Enable exit codes:
               # 2=changes applied
               # 4=failure
    

Software Firewalls, LBs

Install Servers

  • Cobbler
  • MAAS - Ubuntu "Metal As A Service" install server

Orchestration Tools

  • JuJu: mostly for Ubuntu, service orchestration tool (Python, commercially backed)
  • Maestro (enterprise, commercial)
  • mcollective - Puppet parallelizing and orchestration framework
  • SaltStack

Database

Debian

  • Build Kernel Package: How to build kernel packages with make-pkg
    cd /usr/src/linux && make-kpkg clean && make-kpkg --initrd --revision=myrev kernel_image
  • Setup Keyring: How to solve "The following packages cannot be authenticated"
    apt-get install debian-archive-keyring
    apt-get update
  • Force remove broken "reportbug": This can happen during dist-upgrades from Etch/Sarge to Lenny.
  • Packages - Reconfigure after installation:
    dpkg-reconfigure -a
  • dpkg Cheat-Sheet: Query package infos
    # Resolve file to package
    dpkg -S /etc/fstab
    
    # Print all files of a package
    dpkg -L passwd # provided files
    dpkg -c passwd # owned files
    
    # Find packages by name
    dpkg -l gnome*
    
    # Package details
    dpkg -p passwd
    
  • Ubuntu - Access Repositories for older releases. Once a release is deprecated it is moved to old-releases.ubuntu.com. You need to adapt /etc/apt/sources.list to fetch packages from there
    sed -i 's/archive.ubuntu.com/old-releases.ubuntu.com/' /etc/apt/sources.list
  • Ubuntu - List Security Updates
    # Print summary
    /usr/lib/update-notifier/apt-check --human-readable
    
    # Print package names
    /usr/lib/update-notifier/apt-check -p
  • Ubuntu - Upgrade Security Fixes Only
    apt-get dist-upgrade -o Dir::Etc::SourceList=/etc/apt/sources.security.repos.only.list

Debugging / Performance Tools

  • Reboot when /sbin is unusable
    echo b >/proc/sysrq-trigger
  • List Context Switches per Process
    pidstat -w
  • Drop Filesystem Cache
    echo 1 > /proc/sys/vm/drop_caches
  • dmesg - block IO debugging:
    echo 1 > /proc/sys/vm/block_dump
    
    # wait some time...
    echo 0 > /proc/sys/vm/block_dump
    
    # Now check syslog for block dump lines
    
  • Check for changed sysctl() settings:
    sysctl -p
  • dmesg - Filtering Output:
    dmesg -T      # Enable human readable timestamps
    dmesg -x      # Show facility and log level
    dmesg -f daemon     # Filter for facility daemon
    dmesg -l err,crit,alert,emerg # Filter for errors
    
  • lslk - Find file locks: Use lslk to find which PID is blocking an flock() to a file.
  • lsof - Find owners of open file handles:
    lsof      # Complete list
    lsof -i :22    # Filter single TCP port
    lsof -i@192.168.1.5:22 # Filter single connection endpoint
    lsof -u <user>   # Filter per user
    lsof -c <name>   # Filter per process name
    lsof -p 12345    # Filter by PID
    lsof /etc/hosts   # Filter single file
    
  • Perf Tutorial: 2.6+ generic kernel performance statistics tool.
    perf stat -B some_command
  • dstat: Replaces vmstat, iostat, netstat and ifstat and allows to determine PID that is most CPU and most I/O expensive
    dstat -a --top-bio --top-cpu
  • iotop: Python script to monitor I/O like top
  • PHP - How to setup the APD debugger
  • PHP - How to build Debian package for modules from PECL
    apt-get install dh-make-php
    dh-make-pecl <module name>
    cd <source directory>
    debuild
    # .deb package will be in ...
    

Filesystem / Partitioning

  • detox: Tool for recursive cleanup of file names.
    detox -v -r <directory>
  • Fast File Deletion:
    perl -e 'for(<*>){((stat)[9]<(unlink))}'
  • POSIX ACLs:
    getfacl <file>  # List ACLs for file 
    setfacl -m user:joe:rwx dir # Modify ACL
    ls -ld <file>    # Check for active ACL (indicates a "+")
  • uNetBootin: Create bootable media for any distribution. Most useful with USB sticks.
  • Convert ext2 to ext3:
    tune2fs -j /dev/hda1
  • Convert ext3 to ext4:
    tune2fs -O extents,uninit_bg,dir_index /dev/sda1
  • Determine Inode Count:
    tune2fs -l /dev/sda1 | grep Inode
  • Disable ext4 barriers: Add "barrier=0" to the mount options.
  • LVM - Add another disk: How to add a disk to an existing volume
    # Setup partition with (use parted for >2TB)
    (parted) mklabel gpt       # only when >2TB
    (parted) mkpart primary lvm 0 4T    # setup disk full size (e.g. 4TB)
    
    pvcreate /dev/sdb1       # Create physical LVM disk
    vgextend vg01 /dev/sdb1      # Add to volume group
    vgextend -L +4t /dev/mapper/vg01-lvdata  # Extend your volume 
    resize2fs /dev/mapper/vg01-lvdata   # Auto-resize file system
  • rsync - --delete doesn't work: It happens when you call rsync without a trailing slash in the source path like this:
    rsync -az -e ssh --delete /data server:/data

    It just won't delete anything. It will when running it like this:

    rsync -az -e ssh --delete /data/ server:/data

Hosting

  • Hoster Lookup: whoishosthingthis.com, who-hosts.com
  • iplist.net: Simple reverse lookup of neighbour IPs
  • Hoster Status: Status Channels for different hosters:
    • Rackspace:
    • CloudFlare:
    • Hetzner:

Hardware Info

  • HP - Find Installed Memory:
    dmidecode 2>&1 |grep -A17 -i "Memory Device" |egrep "Memory Device|Locator: PROC|Size" |grep -v "No Module Installed" |grep -A1 -B1 "Size:"

Mail

Middleware

  • Heartbeat - Manual IP Failover
    # Either run on the node that should take over
    /usr/share/heartbeat/hb_failover
    
    # Or run on the node to should stop working
    /usr/share/heartbeat/hb_standby
  • keepalived: Simple VRRP solution
  • Pacemaker - Commands
    # Cluster Resource Status
    crm_mon
    crm_mon -1
    crm_mon -f   # failure count
    
    # Dump and Import Config
    cibadmin --query --obj_type resources >file.xml
    cibadmin --replace --obj_type resources --xml-file file.xml
    
    # Resource Handling
    crm resource stop <name>
    crm resource start <name>
    crm resource move <name> <node>
    
    # Put entire cluster in maintenance
    crm configure property maintenance-mode=true
    crm configure property maintenance-mode=false
    
    # Unmanaged Mode for single services
    crm resource unmanage <name>
    crm resource manage <name>
    
  • Pacemaker - Setup Steps
  • RabbitMQ - Commands
    rabbitmqctl list_vhosts   # List all defined vhosts
    rabbitmqctl list_queues <vhost> # List all queues for the vhost
    
    rabbitmqctl report    # Dump detailed report on RabbitMQ instance  
    
    # Plugin management
    /usr/lib/rabbitmq/bin/rabbitmq-plugins enable <name>
    /usr/lib/rabbitmq/bin/rabbitmq-plugins list   
    
  • RabbitMQ - Fix Chef 100% CPU usage
  • RabbitMQ - Setup Clustering
  • wackamole - Commands
    wackatrl -l     # List status
    wackatrl -f     # Remove node from cluster
    wackatrl -s     # Add node to cluster again
    

Monitoring


Network Administration Commands

Package Management

  • Debian File Diversion:
    # Register diverted path and move away
    dpkg-divert --add --rename --divert <renamed file path> &file path>
    
    # Remove a diversion again (remove file first!)
    dpkg-divert --rename --remove <file path>
    
  • Debian
    apt-get install <package> 
    apt-get remove <package> # Remove files installed by <package>
    apt-get purge <package>  # Remove <package> and all the files it did create
    
    apt-get upgrade    # Upgrade all packages
    apt-get install <package> # Upgrade an install package
    
    apt-get dist-upgrade  # Upgrade distribution
    
    apt-cache search <package> # Check if there is such a package name in the repos
    apt-cache clean    # Remove all downloaded .debs
    
    dpkg -l      # List all installed/known packages
    
    # More dpkg invocations above in the "Debian" section!
    
  • Ubuntu (like Debian) with the addition of
    # 1. Edit settings in  /etc/update-manager/release-upgrades
    # e.g. set "Prompt=lts"
    
    # 2. Run upgrade
    do-release-upgrade -d   # For Ubuntu release upgrades
  • Ubuntu: Unattended Upgrades
    apt-get install unattended-upgrades
    dpkg-reconfigure -plow unattended-upgrades 
    # and maybe set notification mail address in /etc/apt/apt.conf.d/50unattended-upgrades
  • OpenSuSE
    zypper install <package> 
    
    zypper refresh    # Update repository infos
    
    zypper list-updates
    zypper repos    # List configured repositories
    
    zypper dist-upgrade   # Upgrade distribution
    zypper dup     # Upgrade distribution (alias)
    
    zypper search <package>  # Search for <package>
    zypper search --search-descriptions <package>
    
    zypper clean      # Clean package cache
    
    # For safe updates:
    zypper mr –keep-packages –remote # Enable caching of packages
    zypper dup -D      # Fetch packages using a dry run
    zypper mr –all –no-refresh  # Set cache usage for following dup
    zypper dup      # Upgrade!
    
  • Redhat:
    up2date
  • Centos:
    yum update     # Upgrade distro
    yum install <package>  # Install <package>

RAID

  • mdadm - Commands
    cat /proc/mdstat   # Print status
    
    mdadm --detail /dev/md0  # Print status per md
    
    mdadm --manage -r /dev/md0 /dev/sda1 # Remove a disk
    mdadm --zero-superblock /dev/sda1  # Initialize a disk
    mdadm --manage -a /dev/md0 /dev/sda1 # Add a disk
    
    mdadm --manage --set-faulty /dev/md0 /dev/sda1
    
  • hpacucli - Commands
    # Show status of all arrays on all controllers
    hpacucli all show config
    hpacucli all show config detail
    
    # Show status of specific controller
    hpacucli ctrl=0 pd all show
    
    # Show Smart Array status
    hpacucli all show status
    
  • LSI MegaRAID - Commands
    # Get number of controllers
    /opt/MegaRAID/MegaCli/MegaCli64 -adpCount -NoLog
    
    # Get number of logical drives on controller #0
    /opt/MegaRAID/MegaCli/MegaCli64 -LdGetNum -a0 -NoLog
    
    # Get info on logical drive #0 on controller #0
    /opt/MegaRAID/MegaCli/MegaCli64 -LdInfo -L0 -a0 -NoLog
    

Security

Shell Scripting - Cheat Sheet

SSH

  • SSH Escape Key: Pressing "~?" (directly following a newline) gives a menu for escape sequences:
    Supported escape sequences:
      ~.  - terminate connection (and any multiplexed sessions)
      ~B  - send a BREAK to the remote system
      ~C  - open a command line
      ~R  - Request rekey (SSH protocol 2 only)
      ~^Z - suspend ssh
      ~#  - list forwarded connections
      ~&  - background ssh (when waiting for connections to terminate)
      ~?  - this message
      ~~  - send the escape character by typing it twice
    (Note that escapes are only recognized immediately after newline.)
    
  • SSH Mounting remote filesystem:
    # To mount a remote home dir 
    sshfs user@server: /mnt/home/user/
    
    # Unmount again with
    fuserumount -u /mnt/home/user
  • authorized_keys HowTo: Syntax and options...
  • Automatic Jump Host Proxying: Use the following ~/.ssh/config snippet and create ~/.ssh/tmp before using it
    ControlMaster auto
    ControlPath /home/<user name>/.ssh/tmp/%h_%p_%r
     
    Host <your jump host>
      ForwardAgent yes
      Hostname <your jump host>
      User <your user name on jump host>
    
    # Note the server list can have wild cards, e.g. "webserver-* database*"
    Host <server list>
      ForwardAgent yes
      User <your user name on all these hosts>
      ProxyCommand ssh -q <your jump host> nc -q0 %h 22
    
  • Easy Key Copying: Stop editing authorized_keys remote. Use the standard OpenSSH ssh-copy-id instead.
    ssh-copy-id [-i keyfile] user@maschine
  • ProxyCommand: Run SSH over a gateway and forward to other hosts based and/or perform some type of authentication. In .ssh/config you can have:
    Host unreachable_host
      ProxyCommand ssh gateway_host exec nc %h %p
  • Transparent Multi-Hop:
    ssh host1 -A -t host2 -A -t host3 ...
  • 100% non-interactive SSH: What parameters to use to avoid any interaction.
    ssh -i my_priv_key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PreferredAuthentications=publickey user@host -n "/bin/ls"
  • SFTP chroot with umask: How to enforce a umask with SFTP
    Subsystem sftp /usr/libexec/openssh/sftp-server -u 0002
  • Agent Forwarding explained with pictures! Configured in /etc/ssh_config with
    Host *
    ForwardAgent yes
  • How to use a SOCKS Proxy On the client start proxy by
    ssh -D <port> <remote host>
  • Parallel SSH on Debian
    apt-get install pssh

    and use it like this

    pssh -h host_list.txt <args>
  • Clustered SSH on Debian
    apt-get install clusterssh

    and use it like this

    cssh server1 server2
  • Vim Remote File Editing:
    vim scp://user@host//some/directory/file.txt

Webserver Stack

Comments

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

To prevent automated spam submissions leave this field empty.
Syndicate content