Dealing with Concurrency with Locks

Automatically Releasing The Lock

Locks are automatically released when their Lock objects are destroyed. This is an implementation detail that is important when sharing Locks between processes. In the example below, pcntl_fork() creates two processes and the Lock will be released automatically as soon as one process finishes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

// ...
$lock = $factory->createLock('pdf-creation');
if (!$lock->acquire()) {
    return;
}

$pid = pcntl_fork();
if (-1 === $pid) {
    // Could not fork
    exit(1);
} elseif ($pid) {
    // Parent process
    sleep(30);
} else {
    // Child process
    echo 'The lock will be released now.';
    exit(0);
}
// ...

To disable this behavior, set the autoRelease argument of LockFactory::createLock() to false. That will make the lock acquired for 3600 seconds or until Lock::release() is called:

1
2
3
4
5

$lock = $factory->createLock(
    'pdf-creation',
    3600, // ttl
    false // autoRelease
);

FlockStore

The FlockStore uses the file system on the local computer to create the locks. It does not support expiration, but the lock is automatically released when the lock object goes out of scope and is freed by the garbage collector (for example when the PHP process ends):

1
2
3
4
5

use Symfony\Component\Lock\Store\FlockStore;

// the argument is the path of the directory where the locks are created
// if none is given, sys_get_temp_dir() is used internally.
$store = new FlockStore('/var/stores');

This store is reliable as long as all concurrent processes use the same physical file system: they must run on the same machine, virtual machine or container and use the same absolute path to the lock directory. Be careful when updating a Kubernetes or Swarm service because, for a short period of time, there can be two containers running in parallel. Be careful also with symlinks that could change at any time (symlink-based release strategies and blue/green deployments often use that trick) and, in general, whenever the path to the lock directory changes between two deployments.

Using a FlockStore in an HTTP context is incompatible with multiple front servers, unless you ensure that the same resource will always be locked on the same machine or you use a well configured shared file system.

Files on the file system can be removed during a maintenance operation (for instance, when cleaning up the /tmp directory or after a reboot of the machine when the directory uses tmpfs). That's not an issue if the lock is released when the process ends, but it is for Lock instances reused between requests.

MemcachedStore

The MemcachedStore saves locks on a Memcached server. It requires a Memcached connection implementing the \Memcached class:

1
2
3
4
5
6

use Symfony\Component\Lock\Store\MemcachedStore;

$memcached = new \Memcached();
$memcached->addServer('localhost', 11211);

$store = new MemcachedStore($memcached);

Memcached stores its items in memory, so locks are not persisted and may disappear by mistake at any time. If the Memcached service or the machine hosting it restarts, every lock is lost without notifying the running processes. To avoid that someone else acquires a lock after a restart, delay the service start for at least as long as the longest lock TTL.

By default, Memcached uses a LRU mechanism to remove old entries when the service needs space to add new items. Keep the number of items stored in Memcached under control; if that's not possible, disable LRU or store the locks in a dedicated Memcached service away from the cache.

MongoDbStore

The MongoDbStore saves locks on a MongoDB server (>= 2.2). It requires a \MongoDB\Collection or \MongoDB\Client from mongodb/mongodb or a MongoDB Connection String:

1
2
3
4
5
6
7
8
9
10
11

use Symfony\Component\Lock\Store\MongoDbStore;

$mongo = 'mongodb://localhost/database?collection=lock';
$options = [
    'gcProbability' => 0.001,
    'database' => 'myapp',
    'collection' => 'lock',
    'uriOptions' => [],
    'driverOptions' => [],
];
$store = new MongoDbStore($mongo, $options);

The MongoDbStore takes the following $options (depending on the first parameter type):

When the first parameter is a:

MongoDB\Collection:

• $options['database'] is ignored
• $options['collection'] is ignored

MongoDB\Client:

• $options['database'] is mandatory
• $options['collection'] is mandatory

MongoDB Connection String:

• $options['database'] is used otherwise /path from the DSN, at least one is mandatory
• $options['collection'] is used otherwise ?collection= from the DSN, at least one is mandatory

The locked resource name is indexed in the _id field of the lock collection. Beware that in MongoDB an indexed field's value can be a maximum of 1024 bytes in length, including the structural overhead.

A TTL index must be used to automatically clean up expired locks. Such an index can be created manually:

1
2
3
4

db.lock.createIndex(
    { "expires_at": 1 },
    { "expireAfterSeconds": 0 }
)

Alternatively, the MongoDbStore::createTtlIndex(int $expireAfterSeconds = 0) method can be called once to create the TTL index during database setup. Read more about Expire Data from Collections by Setting TTL in MongoDB.

This store relies on all PHP application and database nodes having synchronized clocks for lock expiry to occur at the correct time. To ensure locks don't expire prematurely, the lock TTL should be set with enough extra time in expireAfterSeconds to account for any clock drift between nodes.

writeConcern and readConcern are not specified by MongoDbStore, meaning the collection's settings will take effect. readPreference is primary for all queries. Read more about Replica Set Read and Write Semantics in MongoDB.

PdoStore

The PdoStore saves locks in an SQL database. It requires a PDO connection or a Data Source Name (DSN):

1
2
3
4
5

use Symfony\Component\Lock\Store\PdoStore;

// a PDO instance or DSN for lazy connecting through PDO
$databaseConnectionOrDSN = 'mysql:host=127.0.0.1;dbname=app';
$store = new PdoStore($databaseConnectionOrDSN, ['db_username' => 'myuser', 'db_password' => 'mypassword']);

The table where values are stored is created automatically on the first call to the save() method. You can also create this table explicitly by calling the createTable() method in your code.

The PdoStore relies on the ACID properties of the SQL engine. In a cluster configured with multiple primaries, ensure writes are synchronously propagated to every node, or always use the same node. Some SQL engines like MySQL allow disabling the unique constraint check; ensure that this is not the case with SET unique_checks=1;.

In order to purge old locks, this store uses a current datetime to define an expiration date reference. This mechanism relies on all server nodes having synchronized clocks. To ensure locks don't expire prematurely, the TTLs should be set with enough extra time to account for any clock drift between nodes.

DoctrineDbalStore

The DoctrineDbalStore saves locks in an SQL database. It is identical to PdoStore but requires a Doctrine DBAL Connection, or a Doctrine DBAL URL:

1
2
3
4
5

use Symfony\Component\Lock\Store\DoctrineDbalStore;

// a Doctrine DBAL connection or DSN
$connectionOrURL = 'mysql://myuser:mypassword@127.0.0.1/app';
$store = new DoctrineDbalStore($connectionOrURL);

The table where values are stored will be automatically generated when your run the command:

1

$ php bin/console make:migration

If you prefer to create the table yourself and it has not already been created, you can create this table explicitly by calling the createTable() method. You can also add this table to your schema by calling configureSchema() method in your code. If the table has not been created upstream, it will be created automatically on the first call to the save() method.

The same reliability considerations as for the PdoStore apply to this store.

PostgreSqlStore

The PostgreSqlStore uses Advisory Locks provided by PostgreSQL. It requires a PDO connection or a Data Source Name (DSN). It supports native blocking, as well as sharing locks:

1
2
3
4
5

use Symfony\Component\Lock\Store\PostgreSqlStore;

// a PDO instance or DSN for lazy connecting through PDO
$databaseConnectionOrDSN = 'pgsql:host=localhost;port=5634;dbname=app';
$store = new PostgreSqlStore($databaseConnectionOrDSN, ['db_username' => 'myuser', 'db_password' => 'mypassword']);

Unlike the PdoStore, the PostgreSqlStore does not need a table to store locks and the locks do not expire. Instead, locks are automatically released at the end of the session, e.g. in case the client cannot unlock for any reason. Keep in mind that if the PostgreSQL service or the machine hosting it restarts, every lock is lost without notifying the running processes. Likewise, if the TCP connection is lost, PostgreSQL may release locks without notifying the application.

DoctrineDbalPostgreSqlStore

The DoctrineDbalPostgreSqlStore uses Advisory Locks provided by PostgreSQL. It is identical to PostgreSqlStore but requires a Doctrine DBAL Connection or a Doctrine DBAL URL. It supports native blocking, as well as sharing locks:

1
2
3
4
5

use Symfony\Component\Lock\Store\DoctrineDbalPostgreSqlStore;

// a Doctrine Connection or DSN
$databaseConnectionOrDSN = 'postgresql+advisory://myuser:mypassword@127.0.0.1:5634/lock';
$store = new DoctrineDbalPostgreSqlStore($databaseConnectionOrDSN);

The same behavior and reliability considerations as for the PostgreSqlStore apply to this store.

RedisStore

The RedisStore saves locks on a Redis server. It requires a Redis connection implementing the \Redis, \RedisArray, \RedisCluster, \Relay\Relay, \Relay\Cluster or \Predis classes:

1
2
3
4
5
6

use Symfony\Component\Lock\Store\RedisStore;

$redis = new \Redis();
$redis->connect('localhost');

$store = new RedisStore($redis);

Redis stores its items in memory, so locks are not persisted and may disappear by mistake at any time. If the Redis service or the machine hosting it restarts, every lock is lost without notifying the running processes. To avoid that someone else acquires a lock after a restart, delay the service start for at least as long as the longest lock TTL. Redis can also be configured to persist items on disk, but this option slows down writes on the service and could go against other uses of the server.

SemaphoreStore

The SemaphoreStore uses the PHP semaphore functions to create the locks:

1
2
3

use Symfony\Component\Lock\Store\SemaphoreStore;

$store = new SemaphoreStore();

Semaphores are handled by the kernel, so all concurrent processes must run on the same machine, virtual machine or container. Be careful when updating a Kubernetes or Swarm service because, for a short period of time, there can be two containers running in parallel. Before starting a concurrent process on a new machine, check that the other processes are stopped on the old one.

CombinedStore

The CombinedStore is designed for High Availability applications because it manages several stores in sync (for example, several Redis servers). When a lock is acquired, it forwards the call to all the managed stores, and it collects their responses. If a simple majority of stores have acquired the lock, then the lock is considered acquired:

1
2
3
4
5
6
7
8
9
10
11
12
13

use Symfony\Component\Lock\Store\CombinedStore;
use Symfony\Component\Lock\Store\RedisStore;
use Symfony\Component\Lock\Strategy\ConsensusStrategy;

$stores = [];
foreach (['server1', 'server2', 'server3'] as $server) {
    $redis = new \Redis();
    $redis->connect($server);

    $stores[] = new RedisStore($redis);
}

$store = new CombinedStore($stores, new ConsensusStrategy());

Instead of the simple majority strategy (ConsensusStrategy) an UnanimousStrategy can be used to require the lock to be acquired in all the stores:

1
2
3
4

use Symfony\Component\Lock\Store\CombinedStore;
use Symfony\Component\Lock\Strategy\UnanimousStrategy;

$store = new CombinedStore($stores, new UnanimousStrategy());

It's a common mistake to think that this store makes the lock mechanism more reliable. The CombinedStore is, at best, as reliable as the least reliable of all the managed stores: as soon as one managed store returns erroneous information, the CombinedStore won't be reliable. All concurrent processes must use the same configuration, with the same number of managed stores and the same endpoints. Also, instead of using a cluster of Redis or Memcached servers, it's better to use a CombinedStore with a single server per managed store.

ZookeeperStore

The ZookeeperStore saves locks on a ZooKeeper server. It requires a ZooKeeper connection implementing the \Zookeeper class. This store does not support blocking and expiration, but the lock is automatically released when the PHP process is terminated:

1
2
3
4
5
6
7

use Symfony\Component\Lock\Store\ZookeeperStore;

$zookeeper = new \Zookeeper('localhost:2181');
// use the following to define a high-availability cluster:
// $zookeeper = new \Zookeeper('localhost1:2181,localhost2:2181,localhost3:2181');

$store = new ZookeeperStore($zookeeper);

ZooKeeper does not require a TTL because it maintains the locks as ephemeral nodes, which are automatically released at the end of the session (e.g. in case the client cannot unlock for any reason). However, if the ZooKeeper service or the machine hosting it restarts, every lock is lost without notifying the running processes. Also, since the clean up of intermediate nodes becomes an overhead, this store does not support multi-level node locks and all locks are maintained at the root level.

To use ZooKeeper's high-availability feature, you can set up a cluster of multiple servers so that, in case one of the servers goes down, the majority will still be up and serving the requests. All the available servers in the cluster will see the same state.

DynamoDbStore

The DynamoDbStore saves locks on an Amazon DynamoDB table. Install it by running:

1

$ composer require symfony/amazon-dynamo-db-lock

It requires a DynamoDbClient instance or a Data Source Name (DSN):

1
2
3
4
5

use Symfony\Component\Lock\Bridge\DynamoDb\Store\DynamoDbStore;

// a DynamoDbClient instance or DSN
$dynamoDbClientOrDSN = 'dynamodb://default/lock';
$store = new DynamoDbStore($dynamoDbClientOrDSN);

The table where values are stored is created automatically on the first call to the save() method. You can also create this table explicitly by calling the createTable() method in your code.

Remote Stores

Remote stores (see the Scope column in the table of available stores) use a unique token to recognize the true owner of the lock. This token is stored in the Key object and is used internally by the Lock.

Every concurrent process must store the Lock on the same server. Otherwise two different machines may allow two different processes to acquire the same Lock.

Expiring Stores

Expiring stores (see the Expiring column in the table of available stores) guarantee that the lock is acquired only for the defined duration of time. If the task takes longer to be accomplished, then the lock can be released by the store and acquired by someone else.

When you can't be sure that the task will finish within the TTL, the safest solution is to check the remaining lifetime of the lock before each step of the job and handle the unexpected expiration explicitly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

// ...
$lock = $factory->createLock('pdf-creation', 30);

if (!$lock->acquire()) {
    return;
}
while (!$finished) {
    if ($lock->getRemainingLifetime() <= 5) {
        if ($lock->isExpired()) {
            // lock was lost, perform a rollback or send a notification
            throw new \RuntimeException('Lock lost during the overall process');
        }

        $lock->refresh();
    }

    // Perform the task whose duration MUST be less than 5 seconds
}

Choose wisely the lifetime of the Lock and check whether its remaining time to live is enough to perform the task. Keep in mind that storing a Lock usually takes a few milliseconds, but network conditions may increase that time a lot (up to a few seconds); take that into account when choosing the right TTL. In addition, locks are stored on servers with a defined lifetime, so if the date or time of the machine changes, a lock could be released sooner than expected. To guarantee that the date won't change, the NTP service should be disabled and the date should be updated only when the service is stopped.

Changing the Store Configuration

Changing the configuration of stores should be done very carefully. For instance, during the deployment of a new version, processes with the new configuration must not be started while old processes with the old configuration are still running.