When rsync --delete doesn't work

David Grant explains why rsync might silentely refuse to delete anything. It happens when you call rsync without a trailing slash in the source path like this:

rsync -az -e ssh --delete /data server:/data

It just won't delete anything. It will when running it like this:

rsync -az -e ssh --delete /data/ server:/data

Create Random Passwords

In Debian there is a package "mkpasswd" which allows creating passwords like this:

echo | mkpasswd -s

Frame exact splitting with ffmpeg

When preparing videos for Apples HTTP streaming for iPad/iPhone you need to split your video into 10s chunks and provide a play list for Quicktime to process.

The problem lies with frame exact splitting of arbitrary video input material. Wether you split the file using ffmpeg or the Apple segmenter tool you often end up with

  • asynchronous audio in some or all segments
  • missing video frames at the start of each segment
  • audio glitches between two segements
  • missing audio+video between otherwise audio-synchronous consecutive segments

When using the Apple segmenter the only safe way to split files is to convert into an intermediate format which allows frame-exact splitting. As the segmenter only supports transport stream only MPEG-2 TS and MPEG-4 TS do make sense.

To allow frame-exact splitting on problematic input files the easiest way is to blow them up to consist only of I-frames. The parameter for this depends on the output video codec.

An ffmpeg command line for MPEG-2 TS can look like this:

ffmpeg -i inputfile -vcodec mpeg2video -pix_fmt yuv422p -qscale 1 -qmin 1 -intra outputfile

The relevant piece is the "-intra" switch. For MPEG-4 TS something like the following should work:

ffmpeg -i inputfile -vcodec libx264 -vpre slow -vpre baseline -acodec libfaac -ab 128k -ar 44100 -intra -b 2000k -minrate 2000k -maxrate 2000k outputfile

Note: It is important to watch the resulting muxing overhead which might lower the effective bitrate a lot!

The resulting output files should be safe to be passed to the Apple segmenter.

ffmpeg + MT + SVQ3 video = Argh...

Try decoding a video with SVQ3 video codec with multithreading enabled (e.g. -threads 4) ffmpeg r25526 simply refuses to decode it:

Stream #0.0(eng): Video: svq3, yuvj420p, 640x476, 1732 kb/s, 25 fps, 25 tbr, 600 tbn, 600 tbc
...
[svq3 @ 0x806bfe0] SVQ3 does not support multithreaded decoding, patch welcome! (check latest SVN too)
...
Error while opening decoder for input stream #0.0

Instead of simply just using only one thread and just working ffmpeg bails. What a pain.

You need to specify "-threads 1" or no threads option at all for decoding to work.

Linux HTML Rendering Widgets

When you start a new open source project and decide to provide parts or the whole UI by an HTML widget you face the problem of first finding HTML widget libraries, especially light weight ones, and then using the correct one to avoid throwing away the code at a later point when you find one important feature missing.

With this blog post I try to give a summary of existing open source HTML renderer libraries in the Linux world. I have some background experiences with the libraries from working on Liferea where we started with GtkHTML2, later added GtkMozembed support, then added Webkit support and finally switched to WebKit-only rendering.

The following table tries to summarize the simple availability of the different HTML renderers:

Name Toolkit Platform Derived From Driving Force Active
KHTML QT % KDE KDE Yes
wxHtml wxWidgets GTK, Windows KHTML wxWidgets Yes
GtkHtml GTK+ 1.0 GNOME 1 KHTML GNOME 1 No, long gone
GtkHtml2 GTK+ 2.0 GNOME 2 GtkHtml GNOME 2 No, v2.11: Aug 2007
GtkHtml3 GTK+ 2.0 GNOME 2 GtkHtml Ximian, Evolution No, v3.14: May 2008
GtkMozEmbed GTK+ 2.0 Gecko % Mozilla Somewhat
WebKitGtk GTK+ 2.0
GTK+ 3.0
Webkit KHTML Apple Safari Yes

Note: My summary somewhat complements this Wikipedia list. Still it focusses more on Linux renderers and does correctly distinguish between the rather mad history of GtkHtml*.

Given the list above one could conclude the only acceptable renderers are KTHML, wxHtml and WebkitGtk simply based on project activity. Still other renderers like GtkHtml2 and GtkHtml3 have gone a long way and provide a limited but stable functionality.

But the important question is: What features are supported by the different renderers?



Name Widget
Embed
Full
HTML
CSS JS Java/Flash Editor
KHTML y y 1,2,3 y y n
wxHtml y n none n n n
GtkHtml y y none n n y
GtkHtml2 y y 1,2 inline n n n
GtkHtml3 y y none n n y
GtkMozEmbed n y 1,2,3 y y n
WebKitGtk n y 1,2,3 y y n

The feature matrix along with the platform listing explains why a lot of those old renderer libraries are still around. Given you want to render simple markup in an email client you might still choose wxHtml or GtkHtml3, with the latter one providing you with a HTML editor for rich mail editing. Of course when you want to allow your users to have fully fledged inline browsing you need to use either KTHML, GtkMozEmbed or Webkit. Currently I believe WebKitGtk to be the best choice as it's widget gets a lot of attention, which GtkMozEmbed never had while being unstable and rather limited at the same time.

If you find mistakes or have something to add please post a comment!

Comparison of FLV and MP4 metadata tagging tools (injectors)

This post is a comparison of the performance of different tools available to tag FLV and MP4 containers with specific metadata (e.g. title, keyframes, generator or other custom fields...). For FLV containers flvtool2, flvtool++ and yamdi are compared. For the MP4 container MP4box, AtomicParsley and ffmpeg are compared.

Here are the IMO three most important FLV taggers tested on a 125MB FLV:

Name Duration Large Files In Memory Custom Tags Command
flvtool2 1.0.6 3min 11s no no yes flvtool2 -UP -band:Test -user:Test -date:1995 -genres:pop test.flv
flvtool++ 1.2.1 3s no yes yes flvtool++ test.flv -tag band "Test" -tag user "Test" -tag date "1995" -tag genres "pop" test2.flv
yamdi 1.6 1.5s yes no no
(patch)
yamdi -i test.flv -o test2.flv -c "Test"

The performance of flvtool2 is horrendous. For films of 120min it will take hours to process. Therefore: Do not use it! Use Facebooks flvtool++ instead. I guess the bad performance results from it being built in Ruby. Also notice the "Large File" column indicating large file support which officially only yamdi support (by adding compile flag -D_FILE_OFFSET_BITS=64). Another important point is the "In Memory" column indicating that flvtool++ loads the entire file into memory when tagging, which is problematic when tagging large files. Given this results only yamdi should be used for FLV tagging!

Now for the MP4 tagging. Here you can select between a lot of tools from the net, but only a few of them are command line based and available for Unix. The MP4 test file used is 100MB large.

Name Duration Command
AtomicParsely 0.6s AtomicParsley test.mp4 --artist "Test" --genre "Test" --year "1995"
mp4box 0.6s MP4Box -itags Name=Test:Artist=Me:disk=95/100 test.mp4
ffmpeg 0.6 0.8s ffmpeg -i test.mp4 -metadata title="Test" -metadata artist="Test" -metadata date="1995" -acodec copy -vcodec copy test2.mp4

Given that recent ffmpeg brings the tagging for MP4 out of the box (it doesn't for FLV though) you do not even need an external tool to add the metadata,

Making flvtool++ work with large files

The flvtool++ by Facebook is a fast FLV metadata tagger, but at least up to v1.2.1 it lacks large file support. Here is a simple patch to make it work with large files:

--- flvtool++.orig/fout.h	2009-06-19 05:06:47.000000000 +0200
+++ flvtool++/fout.h	2010-10-12 15:51:37.000000000 +0200
@@ -21,7 +21,7 @@
   void open(const char* fn) {
     if (fp) this->close();
 
-    fp = fopen(fn, "wb");
+    fp = fopen64(fn, "wb");
     if (fp == NULL) {
       char errbuf[256];
       snprintf(errbuf, 255, "Error opening output file \"%s\": %s", fn, strerror(errno));

--- flvtool++.orig/mmfile.h	2009-06-19 05:29:43.000000000 +0200
+++ flvtool++/mmfile.h	2010-10-12 15:46:00.000000000 +0200
@@ -16,7 +16,7 @@
 public:
   mmfile() : fd(-1) {} 
   mmfile(char* fn) {
-    fd = open(fn, O_RDONLY);
+    fd = open(fn, O_RDONLY | O_LARGEFILE);
     if (fd == -1) throw std::runtime_error(string("mmfile: unable to open file ") + string(fn));
     struct stat statbuf;
     fstat(fd, &statbuf);

Note: While this patch helps you to process large files flvtool++ will still load the entire file into memory!!! Given this you might want to use a different injector like yamdi. For a
comparsion of existing tools have a look at the Comparison of FLV and MP4 metadata tagging tools.

flvtool2 1.0.6 Bugs

Crash Variant #1

Sometimes flvtool2 1.0.6 crashes on FLVs created by ffmpeg or mencoder. The FLV video itself is playable without the metadata and looks fine, still flvtool2 crashes like this:


$ flvtool2 -kUP -metadatacreator:'some label' video.flv
ERROR: EOFError
ERROR: /usr/lib/ruby/site_ruby/1.8/flv/amf_string_buffer.rb:37:in `read'
ERROR: /usr/lib/ruby/site_ruby/1.8/flv/amf_string_buffer.rb:243:in `read__STRING'
ERROR: /usr/lib/ruby/site_ruby/1.8/flv/audio_tag.rb:56:in `read_header'
ERROR: /usr/lib/ruby/site_ruby/1.8/flv/audio_tag.rb:47:in `after_initialize'
ERROR: /usr/lib/ruby/site_ruby/1.8/flv/tag.rb:56:in `initialize'
ERROR: /usr/lib/ruby/site_ruby/1.8/flv/stream.rb:447:in `new'
ERROR: /usr/lib/ruby/site_ruby/1.8/flv/stream.rb:447:in `read_tags'
ERROR: /usr/lib/ruby/site_ruby/1.8/flv/stream.rb:58:in `initialize'
ERROR: /usr/lib/ruby/site_ruby/1.8/flvtool2/base.rb:272:in `new'
ERROR: /usr/lib/ruby/site_ruby/1.8/flvtool2/base.rb:272:in `open_stream'
ERROR: /usr/lib/ruby/site_ruby/1.8/flvtool2/base.rb:238:in `process_files'
ERROR: /usr/lib/ruby/site_ruby/1.8/flvtool2/base.rb:225:in `each'
ERROR: /usr/lib/ruby/site_ruby/1.8/flvtool2/base.rb:225:in `process_files'
ERROR: /usr/lib/ruby/site_ruby/1.8/flvtool2/base.rb:44:in `execute!'
ERROR: /usr/lib/ruby/site_ruby/1.8/flvtool2.rb:168:in `execute!'
ERROR: /usr/lib/ruby/site_ruby/1.8/flvtool2.rb:228
ERROR: /usr/bin/flvtool2:2:in `require'
ERROR: /usr/bin/flvtool2:2
$

In the Wowza Media Server support forum is a hint on how to patch flvtool2 to solve the issue:

--- /usr/local/lib/site_ruby/1.8/flv/audio_tag.rb	2009-11-12 10:46:13.000000000 +0100
+++ lib/flv/audio_tag.rb	2010-03-17 11:25:35.000000000 +0100
@@ -44,7 +44,9 @@
     
     def after_initialize(new_object)
       @tag_type = AUDIO
-      read_header
+      if data_size > 0
+      	read_header
+      end
     end
 
     def name

Crash Variant #2

Here is another crashing variant:


$ flvtool2 -kUP -metadatacreator:'some label' video.flv
ERROR: EOFError
ERROR: /usr/local/lib/site_ruby/1.8/flv/amf_string_buffer.rb:37:in `read'
ERROR: /usr/local/lib/site_ruby/1.8/flv/amf_string_buffer.rb:75:in `read__AMF_string'
ERROR: /usr/local/lib/site_ruby/1.8/flv/amf_string_buffer.rb:90:in `read__AMF_mixed_array'
ERROR: /usr/local/lib/site_ruby/1.8/flv/amf_string_buffer.rb:134:in `read__AMF_data'
ERROR: /usr/local/lib/site_ruby/1.8/flv/meta_tag.rb:40:in `after_initialize'
ERROR: /usr/local/lib/site_ruby/1.8/flv/tag.rb:56:in `initialize'
ERROR: /usr/local/lib/site_ruby/1.8/flv/stream.rb:451:in `new'
ERROR: /usr/local/lib/site_ruby/1.8/flv/stream.rb:451:in `read_tags'
ERROR: /usr/local/lib/site_ruby/1.8/flv/stream.rb:58:in `initialize'
ERROR: /usr/local/lib/site_ruby/1.8/flvtool2/base.rb:272:in `new'
ERROR: /usr/local/lib/site_ruby/1.8/flvtool2/base.rb:272:in `open_stream'
ERROR: /usr/local/lib/site_ruby/1.8/flvtool2/base.rb:238:in `process_files'
ERROR: /usr/local/lib/site_ruby/1.8/flvtool2/base.rb:225:in `each'
ERROR: /usr/local/lib/site_ruby/1.8/flvtool2/base.rb:225:in `process_files'
ERROR: /usr/local/lib/site_ruby/1.8/flvtool2/base.rb:44:in `execute!'
ERROR: /usr/local/lib/site_ruby/1.8/flvtool2.rb:168:in `execute!'
ERROR: /usr/local/lib/site_ruby/1.8/flvtool2.rb:228
ERROR: /usr/local/bin/flvtool2:2:in `require'
ERROR: /usr/local/bin/flvtool2:2
$

I have not yet found a solution...

Update: I have found crash variant #2 to often happen with larger files. Using flvtool++ instead of flvtool2 always solved the problem. Using flvtool++ is also a good idea as it is much faster than flvtool2. Still both tools have their problems. More about this in the Comparison of FLV and MP4 metadata tagging tools.

Why decoding AAC with ffmpeg doesn't work...

Update: The workaround for the problem doesn't work for ffmpeg versions more recent than 20.06.2011 as libfaad support was dropped in favour of the now stable native ffmpeg AAC encoder! If you still have a separate compilation of libfaad you can workaround using the "faad" encoder tool as described in this post.

If you are using recent ffmpeg versions to decode a .MOV file you might get the following error:

Stream #0.0(eng): Audio: aac, 48000 Hz, 2 channels, s16
Stream #0.1(eng): Video: h264, yuv420p, 1280x530, PAR 1:1 DAR 128:53, 25 tbr, 25 tbn, 50 tbc
Output #0, flv, to 'test.flv':
Stream #0.0(eng): Video: flv (hq), yuv420p, 400x164 [PAR 101:102 DAR 050:2091], 
q=2-31, 300 kb/s, 1k tbn, 25 tbc
Stream #0.1(eng): Audio: libmp3lame, 22050 Hz, 2 channels, s16, 64 kb/s
Stream mapping:
Stream #0.1 -> #0.0
Stream #0.0 -> #0.1
Press [q] to stop encoding
[aac @ 0x80727a0]channel element 1.0 is not allocated
Error while decoding stream #0.0
Error while decoding stream #0.0
Error while decoding stream #0.0
Error while decoding stream #0.0
Error while decoding stream #0.0
Error while decoding stream #0.0
[...]

The message "Error while decoding stream #0.0" is repeated continuously. The resulting video is either unplayable or has no sound. Still the input video is playable in all standard players (VLC, in Windows...). The reason for the problem as I understood it is that the ffmpeg-builtin AAC codec cannot handle an audio stream stream with index "1.0". This is documented in various bugs (see ffmpeg issues #800, #871, #999, #1733...). It doesn't look like this will be handled by ffmpeg very soon. In fact it could well be that they'll handle it as an invalid input file.

Solution: Upgrade to latest ffmpeg and faad library version and add " -acodec libfaad " in front of the "-i" switch. This uses the libfaad AAC decoder, which is said to be a bit slower than the ffmpeg-builtin, but which decodes the AAC without complaining. For example:

ffmpeg -acodec libfaad -i input.mov -b 300kbit/s -ar 22050 -o test.flv

The "-acodec" preceding the "-i" option only influences the input audio decoding, not the audio encoding.

rsync: buffer_append_space: alloc 10512504 not supported

If an rsync call gives you the following:

buffer_append_space: alloc 10512504 not supported
rsync: writefd_unbuffered failed to write 4092 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (36 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(635) [sender=3.0.3]

Don't bother to debug rsync! This is an SSH problem. You need to upgrade your SSH version (which is propably some 4.3 or older).

Syndicate content Syndicate content