Concurrency Control in Go: Inside rqlite’s Custom Synchronization Primitives

submited by

Style Pass

2024-10-02 12:00:03

Concurrency and synchronization are at the heart of any database — and distributed systems too. Today, I’d like to share the set of synchronization primitives and race-free data structures I’ve developed for rqlite, and explain how I use them to program key aspects of the database. They might even be useful to others confronted with similar requirements.

Because of the nature of the Raft consensus algorithm, rqlite periodically needs to perform a process called snapshotting. Snapshotting allows the Raft subsystem to truncate its log, preventing it from growing indefinitely. However, since snapshotting also requires exclusive write access to the database, we want to ensure it never blocks, as this could delay write operations in an unacceptable manner.

Ideally we don’t want snapshotting to start unless we can be confident it can run to completion — so grabbing a simple mutex wouldn’t work. What if the mutex was already held by some other part of the code? What I wanted was the ability to grab a lock, but have the code immediately return if the lock wasn’t available. From a system point-of-view this is OK as Raft will retry the snapshot operation at a later time.