VoltDB is often referred to as an “in-memory” system, but it’s more accurately described as a “memory-first” system. User state has to fit in memory (across the cluster), but there doesn’t have to be any performance tradeoffs to achieve full disk persistence with the performance for which VoltDB is famous.
A snapshot is a complete record of your VoltDB data and application, taken at a single logical point in time, stored on the filesystems of your clustered machines.
What’s a logical point in time? VoltDB offers serializable consistency, meaning every single state can been seen as the serial application of all previous transactions. A point-in-time snapshot represents the state between two transactions, globally, across the whole cluster.
- Snapshots can be periodically scheduled, requested on-demand, or automatically integrated with the command logging feature described below.
- They can include every table, or just a subset.
- They can use a compressed binary format or a more compatible CSV/TSV format.
- Snapshots from production systems can be loaded into testing systems easily, even if the hardware and cluster topology do not match.
- Writing snapshot data to disk is a sequential write, performing well on almost any storage target.
But what about impact? Snapshots put the system in a copy-on-write mode, where the version of data from the snapshot initiation is kept around until safe on disk. This allows VoltDB to persist snapshots up to terabytes in size without impacting ongoing operations.
What about protecting the data mutated between snapshots? VoltDB fills in the gaps between snapshots by writing a log of operations to disk as it processes. In the unlikely event of a cluster-wide failure, the system can be recovered by loading the most recent snapshot, then replaying the log from the snapshot’s logical time to the end of the log. In this way, operations between snapshots can be made safe on disk.
VoltDB’s logging does some pretty clever tricks to avoid the performance problems common with other write-ahead-logs. First, as with snapshots, the VoltDB log is append-only on disk, allowing high throughput on both spinning media and SSDs.
Second, the VoltDB log is logical, rather than binary. This means the database logs the logical description of what it’s going to do, in the order it will be done. Do the same things in the same order to the same starting state, and you end up at the same ending state.
Why logical logging? VoltDB can start to log operations once their order has been determined, which might be well before they are actually processed. In contrast, a binary log can only write to disk after an operation has finished. The difference in latency between the two approaches can be tremendous.
Sync vs. Async Command Logs
Command logging can be performed synchronously or asynchronously.
Synchronous logging requires the data to be on disk (fsynced) before it will be acknowledged as committed to the client. Asynchronous logging allows the disk IO to trail behind committed operations up to a time bound or absolute operation count.
Interestingly, the tradeoffs between data safety, latency and throughput often involve platform choices. Different systems are able to sync data to disk at different rates. Fortunately, modern flash, NVRAM and even flash-backed spinning disk controller can sync at tremendous rates.
Many of VoltDB’s largest and most latency sensitive production deployments use full synchronous disk persistence. These deployments are the system of record, and data safety is paramount. With VoltDB, they don’t have to choose between safety and performance.