Distributed Locks: The Bug That Only Shows Up Under Load

Imagine you're the engineer at a ticketing platform during a major concert drop. Two users click "Book Seat 14B" at the exact same millisecond. Both requests hit different servers. Both servers read the database, see the seat is available, and both write a booking record. Two people just bought the same seat.

Your tests never caught it. Your staging environment never caught it. The bug only showed up when load made two requests truly concurrent. Welcome to the world of distributed systems, where timing is everything and "it worked on my machine" is almost meaningless.

This is the problem distributed locks solve.

Why This Problem Exists

In a single-process application, you can use a mutex or a simple in-memory flag to guard a critical section. One thread acquires the lock, does the work, releases it. Other threads wait. Simple.

In a distributed system, your application runs across multiple servers. There's no shared memory. Each process has its own stack, its own heap, its own mutex. A lock in Process A means nothing to Process B running on a different machine.

Server A                    Server B
   |                           |
   | read: seat available       | read: seat available
   | write: booking             | write: booking
   |                           |
   Both succeed. One seat. Two bookings.

The root cause isn't a bug in your code. It's that distributed systems have no inherent shared state. You need an external coordinator that all nodes respect.

The Analogy: A Hotel Key Card System

Think of a hotel with one master key cabinet behind the front desk. When a guest checks in, the staff physically hands them the key. While that guest holds the key, no one else can get into that room. When they check out, the key goes back.

A distributed lock works the same way. Instead of a key cabinet, you have a shared external store (usually Redis or a database). Any server that wants to perform an exclusive operation must first "check out the key" by writing a lock entry. If the entry already exists, it waits. When done, it deletes the entry, releasing the lock for the next server in line.

The hotel analogy maps precisely: one key exists at a time, the guest holds it exclusively, and returning it makes it available again.

The Solution: Redis-Based Distributed Lock

Redis is the most common choice for distributed locks because of two properties: atomic operations and TTL support. The SET NX PX command lets you set a key only if it doesn't exist, with an expiry, in a single atomic operation.

Wrong: Two separate operations (race condition waiting to happen)

//  Wrong: Check then set is NOT atomic
const exists = await redis.get('lock:seat:14B');
if (!exists) {
  // Another server can sneak in RIGHT HERE
  await redis.set('lock:seat:14B', 'locked');
  // Do the booking...
}

Right: Atomic SET NX with TTL

//  Right: SET NX PX is a single atomic operation
const lockKey = 'lock:seat:14B';
const lockValue = crypto.randomUUID(); // unique per request
const ttlMs = 5000; // lock expires in 5 seconds

const acquired = await redis.set(lockKey, lockValue, 'NX', 'PX', ttlMs);

if (!acquired) {
  // Another process holds the lock — retry or return error
  throw new Error('Seat currently being processed. Try again.');
}

try {
  // Critical section: only ONE server executes this at a time
  await bookSeat('14B', userId);
} finally {
  // Release only if WE still hold the lock (check the value!)
  const currentValue = await redis.get(lockKey);
  if (currentValue === lockValue) {
    await redis.del(lockKey);
  }
}

Notice the lockValue = crypto.randomUUID(). This is critical

Where It Goes Wrong

The Lock Expiry Problem

You set a TTL of 5 seconds. Your booking logic hits a slow database query and takes 8 seconds. The lock expires at 5 seconds. Another server acquires the lock. Now two servers are inside the critical section simultaneously, which is exactly what you were trying to prevent.

Server A acquires lock (TTL: 5s)
  |
  | ... slow DB query ...
  |
[5s] Lock expires automatically
  |
Server B acquires lock (sees it's free)
  |
Server A finishes, tries to release lock
  |
Server A deletes Server B's lock (oops)

This is why you store a unique value per lock acquisition and verify it before deleting. Server A checks: "is the lock value still mine?" If not, Server B already took over, and Server A should not delete B's lock.

The Stale Lock Problem

A server crashes mid-operation and never releases the lock. Without TTL, that lock lives forever and the resource becomes permanently unavailable. TTL is not just a nice-to-have: it's the safety net that keeps a crash from becoming an outage.

The Redis Failover Problem

If your Redis instance goes down between a lock being acquired and released, the lock is gone. If Redis restarts with an empty state (no persistence), all locks vanish. Servers that think they hold a lock are now operating without one.

For most use cases, Redis with AOF persistence is sufficient. For financial-grade scenarios, the Redlock algorithm (using multiple Redis instances) provides stronger guarantees, though it comes with its own trade-offs that Antirez and Martin Kleppmann famously debated in public.

Full Flow Diagram

Client Request
     |
     v
[Try SET NX PX on Redis]
     |
 Acquired?
  /      \
Yes       No
 |         |
 |      Retry / Error
 |
[Execute Critical Section]
 |
[Verify lock value still ours]
 |
[DEL lock key]
 |
[Return response]

Final Thought

Distributed locks feel like a low-level detail until they aren't. Most teams only reach for them after a real incident: duplicate orders, oversold inventory, double-charged payments. The pattern itself isn't complex. What's subtle is all the failure modes hiding inside something that looks like a simple key-value write.

The mental model to carry forward: any time you have a "read, decide, write" sequence that must be atomic across multiple servers, you need coordination. Sometimes that's a database transaction. Sometimes it's an optimistic lock with version numbers. And sometimes, it's a distributed lock with a UUID and a TTL.

If you got the lock, prove it. If you release it, make sure it's still yours.

Have you hit a race condition in production that a distributed lock would have caught? I'd love to hear what the failure looked like and how you debugged it.

#SystemDesign #Backend #Redis #DistributedSystems #Microservices