Cheat Sheets

Is there hope when your Couchbase cluster is stuck in compacting buckets?

Well to be anticlimactic: no.

Scope

This seems to be at least a Couchbase 3.x problem. So far I haven't experienced it with Couchbase 4. Of both versions I only know about the so called community edition.

As for the frequency: Couchbase 3 getting stuck on bucket compacting is propabilistic. In the setups I've run so far it happens every half a year. But this might be load-dependant. Actually never having had the issue on some "smaller" clusters, I actually think it is.

The Symptoms

If you do not monitor explicitly for the compacting status, you will probably noticy by some nodes disks running full. Compacting not working anymore means, the Couchbase disk fragmentation growing and finally filling you disks.

If you look in the GUI you will see a constant "Compacting..." indicator in the top right of the admin GUI. In normal operation it never takes more than some minutes to finish (again depending on your usage).

Things that do not work...

What does help...

The root cause

What actually happened is a data structure corruption from which Couchbase 3 does not recover. This is also the reason why flushing buckets helps.

There are several bug reports in Couchbase 2, 3 and 4 about compacting stuck for different reasons. In general Couchbase is not a very stable product in this regard...

Comment on Disqus