Fix Broken Text Encoding

You have a text file with broken encoding? You want to strip it from all invalid characters?

Here is how to do it:

iconv -c -t ASCII input.txt

The result will be printed to stdout. The -c switch does the stripping. Using -t you can select every target encoding you like.

Shell ANSI Color Matrix

The following script will dump a ANSI color matrix that allows you to easily choose colours. I ripped this off somewhere, just forgot where... This post is for conservation only. Don't read it!


T='gYw'   # The test text

echo -e "\n                 40m     41m     42m     43m\
     44m     45m     46m     47m";

for FGs in '    m' '   1m' '  30m' '1;30m' '  31m' '1;31m' '  32m' \
           '1;32m' '  33m' '1;33m' '  34m' '1;34m' '  35m' '1;35m' \
           '  36m' '1;36m' '  37m' '1;37m';
  do FG=${FGs// /}
  echo -en " $FGs \033[$FG  $T  "
  for BG in 40m 41m 42m 43m 44m 45m 46m 47m;
    do echo -en "$EINS \033[$FG\033[$BG  $T  \033[0m";

Easily fix async video with ffmpeg

1. Correcting Audio that is too slow/fast

This can be done using the "-async" switch of ffmpeg which according to the documentation "Stretches/squeezes" the audio stream to match the timestamps. The parameter takes a numeric value for the samples per seconds to enforce.

Example: ffmpeg -async 25 -i input.mpg <encoding options> -r 25

Try slowly increasing the -async value until audio and video matches.

2. Correcting Time-Shift (Variant 1)

Case 1: Audio ahead of video: As a special case the "-async" switch auto-corrects the start of the audio stream when passed as "-async 1". So try running

ffmpeg -async 1 -i input.mpg <encoding options>

Case 2: Audio behind video: Instead of using "-async" you need to use "-vsync" to drop/duplicate frames in the video stream. There are two methods in the manual page "-vsync 1" and "-vsync 2" and an method auto-detection with "-vsync -1". But using "-map" it is possible to specify the stream to sync against.

Interestingly Google shows people using -aync and -vsync together. So it might be worth experimenting a bit to achieve the intended result :-)

3. Correcting Time-Shift (Variant 2)

If you have a constantly shifted sound/video track that the previous fix doesn't work with, but you know the time shift that needs to be corrected, then you can easily fix it with one of the following two commands:

Case 1: Audio ahead of video:
ffmpeg -i input.flv -itsoffset 00:00:03.0 -i input.flv -vcodec copy -acodec copy -map 0:1 -map 1:0 output_shift3s.flv

Case 2: Audio behind video:
ffmpeg -i input.flv -itsoffset 00:00:03.0 -i input.flv -vcodec copy -acodec copy -map 1:0 -map 0:1 output_shift3s.flv

The difference is in the mapping parameters which specify which of the two supplied input files to map on which output channel. The "-itsoffset" option indicates an offset (3 seconds in the example) for the following input file. The input file is required to have exactly one video channel at position 0 and one audio channel at position 1.

I added "-vcodec copy -acodec copy" to avoid reencoding the video and loose quality. These parameters need to be added after the second input file and before the mapping options. Otherwise one runs into mapping errors.

Update: Also check the comment of an anonymous user below mentioning that he needed a different mapping with a more recent version of ffmpeg. The commands above were tested using ffmpeg 0.5/0.6

When rsync --delete doesn't work

David Grant explains why rsync might silentely refuse to delete anything. It happens when you call rsync without a trailing slash in the source path like this:

rsync -az -e ssh --delete /data server:/data

It just won't delete anything. It will when running it like this:

rsync -az -e ssh --delete /data/ server:/data

Create Random Passwords

In Debian there is a package "mkpasswd" which allows creating passwords like this:

echo | mkpasswd -s

Frame exact splitting with ffmpeg

When preparing videos for Apples HTTP streaming for iPad/iPhone you need to split your video into 10s chunks and provide a play list for Quicktime to process.

The problem lies with frame exact splitting of arbitrary video input material. Wether you split the file using ffmpeg or the Apple segmenter tool you often end up with

  • asynchronous audio in some or all segments
  • missing video frames at the start of each segment
  • audio glitches between two segements
  • missing audio+video between otherwise audio-synchronous consecutive segments

When using the Apple segmenter the only safe way to split files is to convert into an intermediate format which allows frame-exact splitting. As the segmenter only supports transport stream only MPEG-2 TS and MPEG-4 TS do make sense.

To allow frame-exact splitting on problematic input files the easiest way is to blow them up to consist only of I-frames. The parameter for this depends on the output video codec.

An ffmpeg command line for MPEG-2 TS can look like this:

ffmpeg -i inputfile -vcodec mpeg2video -pix_fmt yuv422p -qscale 1 -qmin 1 -intra outputfile

The relevant piece is the "-intra" switch. For MPEG-4 TS something like the following should work:

ffmpeg -i inputfile -vcodec libx264 -vpre slow -vpre baseline -acodec libfaac -ab 128k -ar 44100 -intra -b 2000k -minrate 2000k -maxrate 2000k outputfile

Note: It is important to watch the resulting muxing overhead which might lower the effective bitrate a lot!

The resulting output files should be safe to be passed to the Apple segmenter.

ffmpeg + MT + SVQ3 video = Argh...

Try decoding a video with SVQ3 video codec with multithreading enabled (e.g. -threads 4) ffmpeg r25526 simply refuses to decode it:

Stream #0.0(eng): Video: svq3, yuvj420p, 640x476, 1732 kb/s, 25 fps, 25 tbr, 600 tbn, 600 tbc
[svq3 @ 0x806bfe0] SVQ3 does not support multithreaded decoding, patch welcome! (check latest SVN too)
Error while opening decoder for input stream #0.0

Instead of simply just using only one thread and just working ffmpeg bails. What a pain.

You need to specify "-threads 1" or no threads option at all for decoding to work.

Linux HTML Rendering Widgets

When you start a new open source project and decide to provide parts or the whole UI by an HTML widget you face the problem of first finding HTML widget libraries, especially light weight ones, and then using the correct one to avoid throwing away the code at a later point when you find one important feature missing.

With this blog post I try to give a summary of existing open source HTML renderer libraries in the Linux world. I have some background experiences with the libraries from working on Liferea where we started with GtkHTML2, later added GtkMozembed support, then added Webkit support and finally switched to WebKit-only rendering.

The following table tries to summarize the simple availability of the different HTML renderers:

Name Toolkit Platform Derived From Driving Force Active
wxHtml wxWidgets GTK, Windows KHTML wxWidgets Yes
GtkHtml GTK+ 1.0 GNOME 1 KHTML GNOME 1 No, long gone
GtkHtml2 GTK+ 2.0 GNOME 2 GtkHtml GNOME 2 No, v2.11: Aug 2007
GtkHtml3 GTK+ 2.0 GNOME 2 GtkHtml Ximian, Evolution No, v3.14: May 2008
GtkMozEmbed GTK+ 2.0 Gecko % Mozilla Somewhat
WebKitGtk GTK+ 2.0
GTK+ 3.0
Webkit KHTML Apple Safari Yes

Note: My summary somewhat complements this Wikipedia list. Still it focusses more on Linux renderers and does correctly distinguish between the rather mad history of GtkHtml*.

Given the list above one could conclude the only acceptable renderers are KTHML, wxHtml and WebkitGtk simply based on project activity. Still other renderers like GtkHtml2 and GtkHtml3 have gone a long way and provide a limited but stable functionality.

But the important question is: What features are supported by the different renderers?

Name Widget
CSS JS Java/Flash Editor
KHTML y y 1,2,3 y y n
wxHtml y n none n n n
GtkHtml y y none n n y
GtkHtml2 y y 1,2 inline n n n
GtkHtml3 y y none n n y
GtkMozEmbed n y 1,2,3 y y n
WebKitGtk n y 1,2,3 y y n

The feature matrix along with the platform listing explains why a lot of those old renderer libraries are still around. Given you want to render simple markup in an email client you might still choose wxHtml or GtkHtml3, with the latter one providing you with a HTML editor for rich mail editing. Of course when you want to allow your users to have fully fledged inline browsing you need to use either KTHML, GtkMozEmbed or Webkit. Currently I believe WebKitGtk to be the best choice as it's widget gets a lot of attention, which GtkMozEmbed never had while being unstable and rather limited at the same time.

If you find mistakes or have something to add please post a comment!

Syndicate content