Memcached failover with PHP 5

After having a (lot of) trouble trying to configure Memcached failover under PHP, I thought it would be useful to write down my experience and solutions.

Versions

First, I must specify that I'm running Debian stable, so at the time of writing:
- Memcached: 1.4.13
- libmemcached: 1.0.8
- PHP: 5.4.4
- Memcached extension for PHP: 2.0.1

First test

I'll use a default «$servers» variable, which should represent all your servers, for example:
$servers = array(
array('host1', 11211),
array('host2', 11211));

Important note: you must always use the same order of memcached servers, and you don't have to remove «dead servers»: else your distribution of keys (hash) will change from one script to another, and you'll never perfom with memcached as you should. So set the very same array of servers, in every script you'll use.

After reading several posts, I tried the following setup:
$memcached = new Memcached();
$memcached->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
$memcached->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$memcached->addServers($servers);


Note that «addServers()» must be called after all options are set, else the options won't apply to those servers.

With this setup, a call to a $memcached->set() won't work on the DEAD server, and always return an error. You should be aware that OPT_LIBKETAMA_COMPATIBLE changes the OPT_DISTRIBUTION value, hence it was useless.

Enabling failover

After many other posts, I come up with the following options:
$memcached->setOption(Memcached::OPT_SERVER_FAILURE_LIMIT, 5);
$memcached->setOption(Memcached::OPT_REMOVE_FAILED_SERVERS, 5);

The first one is in fact deprecated since 0.48 version of libmemcached (be careful: there's currently two instance of the variable on the page, the second one shows the deprecation).

The second one is the one you need. But I've found it strange, as it was not working when used as an integer:
- «0»: was returning an error
- «1»: made the failover active after 2 errors
- «2» or more: failover wasn't working anymore

At last, I've found everything on the libmemcached documentation page: it's a boolean value, so either true or false. When you enable it, as a default, the failover starts its job after 2 failures.

I think that the «2 failures» can be configured within libmemcached, but has no setOption() for now.

Final setup

Here's what I've set up on my servers:
$memcached = new Memcached();
$memcached->setOption(Memcached::OPT_CONNECT_TIMEOUT, 10);
$memcached->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
$memcached->setOption(Memcached::OPT_SERVER_FAILURE_LIMIT, 2);
$memcached->setOption(Memcached::OPT_REMOVE_FAILED_SERVERS, true);
$memcached->setOption(Memcached::OPT_RETRY_TIMEOUT, 1);
$memcached->addServers($servers);


Memcached::OPT_DISTRIBUTION: set it to consistent hashing. If one memcached node is dead, its keys (and only its keys) will be evenly distributed to other nodes. This is where the magic is done. This is really different from removing one server in your ->addServers() call.
Memcached::OPT_SERVER_FAILURE_LIMIT: number of connection issues before a server is marked as DEAD, and removed from the list of servers (default: 5).
Memcached::OPT_REMOVE_FAILED_SERVERS: set it to «true», to enable the removal of dead servers.
Memcached::OPT_RETRY_TIMEOUT: after a node is declared DEAD, libmemcached will try it again after that many seconds. Here I've set it to 1 second, but I'm working on PHP scripts that run for less than 100ms most of the time. That would only be useful for cron/daemonize scripts.
Memcached::OPT_CONNECT_TIMEOUT: the timeout after which a server is considered DEAD. As my servers are on the same LAN, ping is ~0.5ms, so 10ms is large enough to consider the server is DEAD. Note that you have to wait twice that time before a node is marked DEAD, so if it's 1000ms, your script will lock for 2 seconds before ignoring the DEAD server. That may affect your response times a lot, and that's why I've set it very low.

Photo by Bengt Nyman.