Following the recent post on how to investigate limit related issues which gave instructions what to check if you suspect a system limit to be hit I want to share this Nagios check to cover the open file descriptor limit. Note that existing Nagios plugins like this only check the global limit, only check one application or do not output all problems. So here is my solution which does:

1. Check the global file descriptor limit
2. Uses lsof to check all processes "nofile" hard limit

It has two simple parameters -w and -c to specify a percentage threshold. An example call:

./check_nofile_limit.sh -w 70 -c 85

could result in the following output indicating two problematic processes:

WARNING memcached (PID 2398) 75% of 1024 used CRITICAL apache (PID 2392) 94% of 4096 used

Here is the check script doing this:

#!/bin/bash

# Check "nofile" limit for all running processes using lsof

MIN_COUNT=0	# default "nofile" limit is usually 1024, so no checking for 
		# processes with much less open fds needed

WARN_THRESHOLD=80	# default warning:  80% of file limit used
CRITICAL_THRESHOLD=90	# default critical: 90% of file limit used

while getopts "hw:c:" option; do
	case $option in
		w) WARN_THRESHOLD=$OPTARG;;
		c) CRITICAL_THRESHOLD=$OPTARG;;	
		h) echo "Syntax: $0 [-w <warning percentage>] [-c <critical percentage>]"; exit 1;;
	esac
done

results=$(
# Check global limit
global_max=$(cat /proc/sys/fs/file-nr 2>&1 |cut -f 3)
global_cur=$(cat /proc/sys/fs/file-nr 2>&1 |cut -f 1)
ratio=$(( $global_cur * 100 / $global_max))

if [ $ratio -ge $CRITICAL_THRESHOLD ]; then
	echo "CRITICAL global file usage $ratio% of $global_max used"
elif [ $ratio -ge $WARN_THRESHOLD ]; then
	echo "WARNING global file usage $ratio% of $global_max used"
fi

# We use the following lsof options:
#
# -n 	to avoid resolving network names
# -b	to avoid kernel locks
# -w	to avoid warnings caused by -b
# +c15	to get somewhat longer process names
#
lsof -wbn +c15 2>/dev/null | awk '{print $1,$2}' | sort | uniq -c |\
while read count name pid remainder; do
	# Never check anything above a sane minimum
	if [ $count -gt $MIN_COUNT ]; then
		# Extract the hard limit from /proc
		limit=$(cat /proc/$pid/limits 2>/dev/null| grep 'open files' | awk '{print $5}')

		# Check if we got something, if not the process must have terminated
		if [ "$limit" != "" ]; then
			ratio=$(( $count * 100 / $limit ))
			if [ $ratio -ge $CRITICAL_THRESHOLD ]; then
				echo "CRITICAL $name (PID $pid) $ratio% of $limit used"
			elif [ $ratio -ge $WARN_THRESHOLD ]; then
				echo "WARNING $name (PID $pid) $ratio% of $limit used"
			fi
		fi
	fi
done
)

if echo $results | grep CRITICAL; then
	exit 2
fi
if echo $results | grep WARNING; then
	exit 1
fi

echo "All processes are fine."

Use the script with caution! At the moment it has no protection against a hanging lsof. So the script might mess up your system if it hangs for some reason. If you have ideas how to improve it please share them in the comments!

This is a check list of all you can do wrong when trying to set limits on Debian/Ubuntu. The hints might apply to other distros too, but I didn't check. If you have additional suggestions please leave a comment!

Always Check Effective Limit

The best way to check the effective limits of a process is to dump

/proc/<pid>/limits

which gives you a table like this

Limit  Soft Limit Hard Limit Units 
Max cpu time unlimited unlimited seconds 
Max file size unlimited unlimited bytes 
Max data size unlimited unlimited bytes 
Max stack size 10485760 unlimited bytes 
Max core file size 0 unlimited bytes 
Max resident set unlimited unlimited bytes 
Max processes 528384 528384 processes 
Max open files 1024 1024 files 
Max locked memory 32768 32768 bytes 
Max address space unlimited unlimited bytes 
Max file locks unlimited unlimited locks 
Max pending signals 528384 528384 signals 
Max msgqueue size 819200 819200 bytes 
Max nice priority 0 0 
Max realtime priority 0 0 

Running "ulimit -a" in the shell of the respective user rarely tells something because the init daemon responsible for launching services might be ignoring /etc/security/limits.conf as this is a configuration file for PAM only and is applied on login only per default.

Do Not Forget The OS File Limit

If you suspect a limit hit on a system with many processes also check the global limit:

$ cat /proc/sys/fs/file-nr
7488	0	384224
$

The first number is the number of all open files of all processes, the third is the maximum. If you need to increase the maximum:

# sysctl -w fs.file-max=500000

Ensure to persist this in /etc/sysctl.conf to not loose it on reboot.

Check "nofile" Per Process

Just checking the number of files per process often helps to identify bottlenecks. For every process you can count open files from using lsof:

lsof -n -p <pid> | wc -l

So a quick check on a burning system might be:

lsof -n 2>/dev/null | awk '{print $1 " (PID " $2 ")"}' | sort | uniq -c | sort -nr | head -25

whic returns the top 25 file descriptor eating processes

 139 mysqld (PID 2046)
 105 httpd2-pr (PID 25956)
 105 httpd2-pr (PID 24384)
 105 httpd2-pr (PID 24377)
 105 httpd2-pr (PID 24301)
 105 httpd2-pr (PID 24294)
 105 httpd2-pr (PID 24239)
 105 httpd2-pr (PID 24120)
 105 httpd2-pr (PID 24029)
 105 httpd2-pr (PID 23714)
 104 httpd2-pr (PID 3206)
 104 httpd2-pr (PID 26176)
 104 httpd2-pr (PID 26175)
 104 httpd2-pr (PID 26174)
 104 httpd2-pr (PID 25957)
 104 httpd2-pr (PID 24378)
 102 httpd2-pr (PID 32435)
 53 sshd (PID 25607)
 49 sshd (PID 25601)

The same more comfortable including the hard limit:

lsof -n 2>/dev/null | awk '{print $1,$2}' | sort | uniq -c | sort -nr | head -25 | while read nr name pid ; do printf "%10d / %-10d %-15s (PID %5s)\n" $nr $(cat /proc/$pid/limits | grep 'open files' | awk '{print $5}') $name $pid; done

returns

 105 / 1024 httpd2-pr (PID 5368)
 105 / 1024 httpd2-pr (PID 3834)
 105 / 1024 httpd2-pr (PID 3407)
 104 / 1024 httpd2-pr (PID 5392)
 104 / 1024 httpd2-pr (PID 5378)
 104 / 1024 httpd2-pr (PID 5377)
 104 / 1024 httpd2-pr (PID 4035)
 104 / 1024 httpd2-pr (PID 4034)
 104 / 1024 httpd2-pr (PID 3999)
 104 / 1024 httpd2-pr (PID 3902)
 104 / 1024 httpd2-pr (PID 3859)
 104 / 1024 httpd2-pr (PID 3206)
 102 / 1024 httpd2-pr (PID 32435)
 55 / 1024 mysqld (PID 2046)
 53 / 1024 sshd (PID 25607)
 49 / 1024 sshd (PID 25601)
 46 / 1024 dovecot-a (PID 1869)
 42 / 1024 python (PID 1850)
 41 / 1048576 named (PID 3130)
 40 / 1024 su (PID 25855)
 40 / 1024 sendmail (PID 3172)
 40 / 1024 dovecot-a (PID 14057)
 35 / 1024 sshd (PID 3160)
 34 / 1024 saslauthd (PID 3156)
 34 / 1024 saslauthd (PID 3146)

Upstart doesn't care about limits.conf!

The most common mistake is believing upstart behaves like the Debian init script handling. When on Ubuntu a service is being started by upstart /etc/security/limits.conf will never apply! To get upstart to change the limits of a managed service you need to insert a line like

limit nofile 10000 20000

into the upstart job file in /etc/init.

When Changing /etc/security/limits.conf Re-Login!

After you apply a change to /etc/security/limits.conf you need to login again to have the change applied to your next shell instance by PAM. Alternatively you can use sudo -i to switch to user whose limits you modified and simulate a login.

Special Debian Apache Handling

The Debian Apache package which is also included in Ubuntu has a separate way of configuring "nofile" limits. If you run the default Apache in 12.04 and check /proc/<pid>/limits of a Apache process you'll find it is allowing 8192 open file handles. No matter what you configured elsewhere.

This is because Apache defaults to 8192 files. If you want another setting for "nofile" then you need to edit /etc/apache2/envvars.

For Emergencies: prlimit!

Starting with util-linux-2.21 there will be a new "prlimit" tool which allows you to easily get/set limits for running processes. Sadly Debian is and will be for some time on util-linux-2.20. So what do we do in the meantime?

The prlimit(2) manpage which is for the system call prlimit() gives a hint: at the end of the page there is a code snippet to change the CPU time limit. You can adapt it to any limit you want by replacing RLIMIT_CPU with any of

• RLIMIT_NOFILE
• RLIMIT_OFILE
• RLIMIT_AS
• RLIMIT_NPROC
• RLIMIT_MEMLOCK
• RLIMIT_LOCKS
• RLIMIT_SIGPENDING
• RLIMIT_MSGQUEUE
• RLIMIT_NICE
• RLIMIT_RTPRIO
• RLIMIT_RTTIME
• RLIMIT_NLIMITS

You might want to check "/usr/include/$(uname -i)-linux-gnu/bits/resource.h". Check the next section for an ready made example for "nofile".

Build Your Own set_nofile_limit

The per-process limit most often hit is propably "nofile". Imagine you production database suddenly running out of files. Imagine a tool that can instant-fix it without restarting the DB!

Copy the following code to a file "set_limit_nofile.c"

#define _GNU_SOURCE
#define _FILE_OFFSET_BITS 64
#include <stdio.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/resource.h>

#define errExit(msg) do { perror(msg); exit(EXIT_FAILURE); \
 } while (0)

int
main(int argc, char *argv[])
{
 struct rlimit old, new;
 struct rlimit *newp;
 pid_t pid;

 if (!(argc == 2 || argc == 4)) {
 fprintf(stderr, "Usage: %s <pid> [<new-soft-limit> "
 "<new-hard-limit>]\n", argv[0]);
 exit(EXIT_FAILURE);
 }

 pid = atoi(argv[1]); /* PID of target process */

 newp = NULL;
 if (argc == 4) {
 new.rlim_cur = atoi(argv[2]);
 new.rlim_max = atoi(argv[3]);
 newp = &new;
 }

 if (prlimit(pid, RLIMIT_NOFILE, newp, &old) == -1)
 errExit("prlimit-1");
 printf("Previous limits: soft=%lld; hard=%lld\n",
 (long long) old.rlim_cur, (long long) old.rlim_max);

 if (prlimit(pid, RLIMIT_NOFILE, NULL, &old) == -1)
 errExit("prlimit-2");
 printf("New limits: soft=%lld; hard=%lld\n",
 (long long) old.rlim_cur, (long long) old.rlim_max);

 exit(EXIT_FAILURE);
}

and compile it with

gcc -o set_nofile_limit set_nofile_limit.c

And now you have a tool to change any processes "nofile" limit. To dump the limit just pass a PID:

$ ./set_limit_nofile 17006
Previous limits: soft=1024; hard=1024
New limits: soft=1024; hard=1024
$

To change limits pass PID and two limits:

# ./set_limit_nofile 17006 1500 1500
Previous limits: soft=1024; hard=1024
New limits: soft=1500; hard=1500
# 

And the production database is saved.

When on Ubuntu setting ulimits is strange enough as upstart does ignore /etc/security/limits.conf. You need to use the "limit" stanza to change any limit.

But try it with Apache and it won't help as Debian invented another way to ensure Apache ulimits can be changed. So you need to always check

/etc/apache2/envvars

which in a fresh installation contains a line

#APACHE_ULIMIT_MAX_FILES="ulimit -n 65535"

Uncomment it to set any max file limit you want. Restart Apache and verify the process limit in /proc/<pid>/limits which should give you something like

$ cat /proc/3643/limits
Limit Soft Limit Hard Limit Units 
Max cpu time unlimited unlimited seconds 
Max file size unlimited unlimited bytes 
Max data size unlimited unlimited bytes 
Max stack size 8388608 unlimited bytes 
Max core file size 0 unlimited bytes 
Max resident set unlimited unlimited bytes 
Max processes 30404 30404 processes 
Max open files 1024 4096 files <<<--------------
Max locked memory 65536 65536 bytes 
Max address space unlimited unlimited bytes 
Max file locks unlimited unlimited locks 
Max pending signals 30404 30404 signals 
Max msgqueue size 819200 819200 bytes 
Max nice priority 0 0 
Max realtime priority 0 0 
Max realtime timeout unlimited unlimited us 
$

Problem:

merb : chef-server (api) : worker (port 4000) ~ Connection refused - connect(2) - (Errno::ECONNREFUSED)

Solution: Check why solr is not running and start it

/etc/init.d/chef-solr start

Problem:

merb : chef-server (api) : worker (port 4000) ~ Net::HTTPFatalError: 503 "Service Unavailable" - (Chef::Exceptions::SolrConnectionError)

Solution: You need to check solr log for error. You can find

• the access log in /var/log/chef/2013_03_01.jetty.log (adapt the date)
• the solr error log in /var/log/chef/solr.log

Hopefully you find an error trace there.

Problem:

# chef-expander -n 1
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- http11_client (LoadError)
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/vendor_ruby/em-http.rb:8:in `'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/vendor_ruby/em-http-request.rb:1:in `'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/vendor_ruby/chef/expander/solrizer.rb:24:in `'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/vendor_ruby/chef/expander/vnode.rb:26:in `'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/vendor_ruby/chef/expander/vnode_supervisor.rb:28:in `'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/vendor_ruby/chef/expander/cluster_supervisor.rb:25:in `'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /usr/bin/chef-expander:27:in `
'

Solution: This is a gems dependency issue with the HTTP client gem. Read about it here: http://tickets.opscode.com/browse/CHEF-3495. You might want to check the active Ruby version you have on your system e.g. on Debian run

update-alternatives --config ruby

to find out and change it. Note that the emhttp package from Opscode might require a special version. You can check by listing the package files:

dpkg -L libem-http-request-ruby
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libem-http-request-ruby
/usr/share/doc/libem-http-request-ruby/changelog.Debian.gz
/usr/share/doc/libem-http-request-ruby/copyright
/usr/lib
/usr/lib/ruby
/usr/lib/ruby/vendor_ruby
/usr/lib/ruby/vendor_ruby/em-http.rb
/usr/lib/ruby/vendor_ruby/em-http-request.rb
/usr/lib/ruby/vendor_ruby/em-http
/usr/lib/ruby/vendor_ruby/em-http/http_options.rb
/usr/lib/ruby/vendor_ruby/em-http/http_header.rb
/usr/lib/ruby/vendor_ruby/em-http/client.rb
/usr/lib/ruby/vendor_ruby/em-http/http_encoding.rb
/usr/lib/ruby/vendor_ruby/em-http/multi.rb
/usr/lib/ruby/vendor_ruby/em-http/core_ext
/usr/lib/ruby/vendor_ruby/em-http/core_ext/bytesize.rb
/usr/lib/ruby/vendor_ruby/em-http/mock.rb
/usr/lib/ruby/vendor_ruby/em-http/decoders.rb
/usr/lib/ruby/vendor_ruby/em-http/version.rb
/usr/lib/ruby/vendor_ruby/em-http/request.rb
/usr/lib/ruby/vendor_ruby/1.8
/usr/lib/ruby/vendor_ruby/1.8/x86_64-linux
/usr/lib/ruby/vendor_ruby/1.8/x86_64-linux/em_buffer.so
/usr/lib/ruby/vendor_ruby/1.8/x86_64-linux/http11_client.so

The listing above for example indicates ruby1.8.

Munin tends to spam the log files it writes (munin-update.log, munin-graph.log...) with many lines at INFO log level. It also doesn't respect syslog log levels as it uses Log4perl.

In difference to the munin-node configuration (munin-node.conf) there is no "log_level" setting in the munin.conf at least in the versions I'm using right now.

So let's fix the code. In /usr/share/munin/munin-<update|graph> find the following lines:

logger_open($config->{'logdir'});
logger_debug() if $config->{debug} or defined($ENV{CGI_DEBUG});

and change it to

logger_open($config->{'logdir'});
logger_debug() if $config->{debug} or defined($ENV{CGI_DEBUG});
logger_level("warn"); 

As parameter to logger_level() you can provide "debug", "info", "warn", "error" or "fatal" (see manpage "Munin::Master::Logger".

And finally: silent logs!

Here is a short summary on what not to do wrong when using strpos() and what other matching methods do exist in PHP.

Important: Use the Correct Comparison Operator!

Most PHP beginners fall for this:

if (strpos($str, $substr) != false) {

if (strpos($str, $substr) == false) {

if (strpos($str, $substr) === false) {

if (strpos($str, $substr) !== false) {

The reason for using the identical operators (that also check for the type of the value) "===" or "!==" is that strpos() might return "0" for match at the beginning or "false" for no match which cannot be distinguished by "==" or "!=".

Simple rule: With strpos() functions always compare to "false" with the identical operators.

Case Sensitivity

It is simple:

Search From Offset

If you want to continue a search or want to skip something at the start use an offset as third parameter:

if (strpos($str, $substr, 5) === false) {

By using the return value of strpos() you can do repeated searches:

$pos1 = strpos($str, $substr);
$pos2 = strpos($str, $substr, $pos1 + strlen($pos1));

Search Backwards

There is an additional set of methods for backwards searching:

Extract Found Strings

If you want to extract a string at the position you've found you need to use the substr() method.

An example:

$pos = strpos($str, $substr);
$result = substr($str, $pos);

When not to use strpos()?

Do not use strpos() if you want to do the following things:

1. Test for a pattern at the start or end of a string -> use preg_match()
2. Extract multiple substrings -> use preg_match()
3. Match a string for a complex pattern -> use preg_match()
4. If you want to split a string a delimiters -> use strpbrk()

About Performance

If you are wondering how fast strpos() is compared to using regular expressions check this post: strpos() vs preg_match()

Imagine a legacy script. Very long, complex and business critical. No time for a rewrite and no fixed requirements. But you want to have timestamps added to the output it produces.

Search and Replace

One way to do this is find each and every echo and replace it.

echo "Doing X with $file."

becomes

log "Doing X with $file."

and you implement log() as a function prefixing the timestamp. The danger here is to not replace some echo that needs being redirected somewhere else.

Alternative: Wrap in a Subshell

Instead of modifying all echo's one could do the following and just "wrap" the whole script:

#!/bin/bash

$(

<the original script body>

) | stdbuf -i0 -o0 -e0 awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; }'

You can drop the stdbuf invocation that unbuffers the pipe if you do not need 100% exact timestamps. Of course you can use this also with any slow running command on the shell no matter how complex, just add the pipe command.

The obvious advantage: you do not touch the legacy code and can be 100% sure the script is still working after adding the timestamps.

Search for chef 100% cpu issue and you will find a lot of sugestions ranging from reboot the server, to restart RabbitMQ and often to check the kernel max file limit.

All of those do not help! What does help is checking RabbitMQ with

rabbitmqctl report | grep -A3 file_descriptors

and have a look at the printed limits and usage. Here is an example:

 {file_descriptors,[{total_limit,8900},
                    {total_used,1028},
                    {sockets_limit,8008},
                    {sockets_used,2}]},

In my case the 100% CPU usage was caused by all of the file handles being used up which for some reason causes RabbitMQ 2.8.4 to go into a crazy endless loop rarely responding at all.

The "total_limit" value is the "nofile" limit for the maximum number of open files you can check using "ulimit -n" as RabbitMQ user. Increase it permanently by defining a RabbitMQ specific limit for example in /etc/security/limits.d/rabbitmq.conf:

rabbitmq    soft   nofile   10000

or using for example

ulimit -n 10000

from the start script or login scripts. Then restart RabbitMQ. The CPU usage should be gone.

Update: This problem only affects RabbitMQ releases up to 1.8.4 and should be fixed starting with 1.8.5.

When watching other people familiar with Linux type commands on a bash shell what one often needs is a lot of patience. It seems that many of us (and I include myself) do not know or regularily forget about the biggest timesavers.

Here is a list of maybe trivial things I consider absolutely helpful to efficiently use Bash in Emacs mode.

1. Jump Words not Characters!

When scrolling through a long command: save time by holding Ctrl to skip words. Alternatively press ESC before each cursor press or use Alt-B and Alt-F.

2. Kill Words not Characters!

The same when deleting stuff. Don't press Backspace 30 times. Use a hotkey to do it in one step:

• Alt-Backspace or Ctrl-w to delete a word backwards (the difference lies in delimiters)
• Alt-d to delete a word from start

3. Lookup Filenames while Writing

Do not cancel writing commands just because you forgot a path somewhere on the system. Look it up while writing the command. If paths are in quotes, temporarily disable the quotes to make expansion work again.

4. Save Cancelled Commands

If you still decide to cancel a command because you want to look something up, then keep it in the history. Just type a "#" in front of it and Enter to safely store it for later reference.

5. Use Ctrl-A and Ctrl-E

Most beginners do scroll around a lot for no good reason. This hurts on slow connections. Do never scroll to the start/end of the command. Use Ctrl-a for start of line and Ctrl-E for end of line. When in screen use Ctrl-a-a instead of Ctrl-a!

6. Do not overuse Ctrl-R!

Many users heavily rely on Ctrl-R for reverse history search and use it to go back >50 times to find a command of 10 characters! Just type the 10 characters! Or if you thought it was only 10 steps away in the history just cancel when you do not find it early on. This is mostly a psychological thing: "it might be around the corner" but wastes a lot of time.

When your command is only a few steps away in the history use the Up/Down leys until you find it. Cancel this once you press as many times as your command is long!

7. Use Undo!

Avoid rewriting stuff if you mess up.

Case 1: If you delete something and want to restore it use Ctrl+_ (Ctrl+Shift+Dash) to undo the last change.

Case 2:If you have edited a history entry you got using Ctrl-R or Up/Down navigation and want to restore it to it's original value use Alt-R. In general try to avoid editing history commands as this is confusing!

8. Use History with Line Numbers

Often I see people calling "history" and then mouse copy&pasting a command the find. The faster thing to do is to type "!<number>" to execute a specific command. Usually just 4-5 characters eliminating the possibility of pasting the wrong copy buffer or a command with line breaks after a lot of fiddling with the mouse.

What Else?

There might be other useful commands, but I think I've listed the time savers above. Feel free to comment your suggestions!

When trying to implement a media plugin for Liferea I learned at lot from Laszlo Pandy's session slides from the Ubuntu Opportunistic Developer Week (PDF, source). The only problem was that it is for PyGtk, the GTK binding for Python, which is now more or less deprecated for PyGI, the GTK+ 3.0 GObject introspection based binding.

While it is easy to convert all pieces manually following the Novacut/GStreamer1.0 documentation I still want to share a complete music player source example that everyone interested can copy from. I hope this saves the one or the other some time in guessing how to write something like "gst.FORMAT_TIME" in PyGI (actually it is "Gst.Format.TIME").

So here is the example code (download file):

from gi.repository import GObject
from gi.repository import GLib
from gi.repository import Gtk
from gi.repository import Gst

class PlaybackInterface:

    def __init__(self):
	self.playing = False

	# A free example sound track
	self.uri = "http://cdn02.cdn.gorillavsbear.net/wp-content/uploads/2012/10/GORILLA-VS-BEAR-OCTOBER-2012.mp3"

	# GTK window and widgets
	self.window = Gtk.Window()
	self.window.set_size_request(300,50)

	vbox = Gtk.Box(Gtk.Orientation.HORIZONTAL, 0)
	vbox.set_margin_top(3)
	vbox.set_margin_bottom(3)
	self.window.add(vbox)

	self.playButtonImage = Gtk.Image()
	self.playButtonImage.set_from_stock("gtk-media-play", Gtk.IconSize.BUTTON)
	self.playButton = Gtk.Button.new()
	self.playButton.add(self.playButtonImage)
	self.playButton.connect("clicked", self.playToggled)
	Gtk.Box.pack_start(vbox, self.playButton, False, False, 0)

	self.slider = Gtk.HScale()
	self.slider.set_margin_left(6)
	self.slider.set_margin_right(6)
	self.slider.set_draw_value(False)
	self.slider.set_range(0, 100)
	self.slider.set_increments(1, 10)

	Gtk.Box.pack_start(vbox, self.slider, True, True, 0)

	self.label = Gtk.Label(label='0:00')
	self.label.set_margin_left(6)
	self.label.set_margin_right(6)
	Gtk.Box.pack_start(vbox, self.label, False, False, 0)

	self.window.show_all()

        # GStreamer Setup
        Gst.init_check(None)
        self.IS_GST010 = Gst.version()[0] == 0
	self.player = Gst.ElementFactory.make("playbin2", "player")
	fakesink = Gst.ElementFactory.make("fakesink", "fakesink")
	self.player.set_property("video-sink", fakesink)
	bus = self.player.get_bus()
	#bus.add_signal_watch_full()
	bus.connect("message", self.on_message)
	self.player.connect("about-to-finish",  self.on_finished)

    def on_message(self, bus, message):
	t = message.type
	if t == Gst.Message.EOS:
		self.player.set_state(Gst.State.NULL)
		self.playing = False
	elif t == Gst.Message.ERROR:
		self.player.set_state(Gst.State.NULL)
		err, debug = message.parse_error()
		print "Error: %s" % err, debug
		self.playing = False

	self.updateButtons()

    def on_finished(self, player):
	self.playing = False
        self.slider.set_value(0)
	self.label.set_text("0:00")
        self.updateButtons()

    def play(self):
	self.player.set_property("uri", self.uri)
	self.player.set_state(Gst.State.PLAYING)
	GObject.timeout_add(1000, self.updateSlider)

    def stop(self):
	self.player.set_state(Gst.State.NULL)
	
    def playToggled(self, w):
        self.slider.set_value(0)
	self.label.set_text("0:00")

	if(self.playing == False):
		self.play()
	else:
		self.stop()

	self.playing=not(self.playing)
	self.updateButtons()

    def updateSlider(self):
	if(self.playing == False):
	   return False # cancel timeout

	try:
	   if self.IS_GST010:
	      nanosecs = self.player.query_position(Gst.Format.TIME)[2]
	      duration_nanosecs = self.player.query_duration(Gst.Format.TIME)[2]
	   else:
	      nanosecs = self.player.query_position(Gst.Format.TIME)[1]
	      duration_nanosecs = self.player.query_duration(Gst.Format.TIME)[1]

	   # block seek handler so we don't seek when we set_value()
	   # self.slider.handler_block_by_func(self.on_slider_change)

           duration = float(duration_nanosecs) / Gst.SECOND
	   position = float(nanosecs) / Gst.SECOND
	   self.slider.set_range(0, duration)
	   self.slider.set_value(position)
           self.label.set_text ("%d" % (position / 60) + ":%02d" % (position % 60))

	   #self.slider.handler_unblock_by_func(self.on_slider_change)
	
	except Exception as e:
		# pipeline must not be ready and does not know position
		print e
		pass

	return True

    def updateButtons(self):
        if(self.playing == False):
           self.playButtonImage.set_from_stock("gtk-media-play", Gtk.IconSize.BUTTON)
        else:
           self.playButtonImage.set_from_stock("gtk-media-stop", Gtk.IconSize.BUTTON)

if __name__ == "__main__":
    PlaybackInterface()
    Gtk.main()

and this is how it should look like if everything goes well:

Please post comments below if you have improvement suggestions!