VoltDB is commonly used as an event processing system. Something happened, and you want to react to it. So far so good.
In reality, your application may be trying to process many events at the same time, often tens or hundreds of thousands per second. Furthermore, hardware fails and networks hiccup.
In this environment, how can you be sure you didn’t miss some events, or process some events more than once? It turns out that this isn’t just hard; it’s impossible! For some deeper background, you might want check out Tyler Treat’s blog post on the topic.
The good news is that you can fake it with a system like VoltDB.
Pick At-Least-Once or At-Most-Once
It all comes down to the event source, often called the producer or the client.
- The source can send every event once, which ensures that no event will be processed twice. We call this At-Most-Once delivery.
The source can also resend events until an acknowledgement is received, ensuring the event gets processed. We call this At-Least-Once delivery.
- There are plenty of systems that make neither guarantee, but picking one of these is the best you can do.
Idempotence is a property of an operation that says if you apply the operation multiple times, it doesn’t have any different effects than if it was applied once. Check out the Wikipedia page.
- Round a decimal number to the nearest whole number
- Correct misspellings in some text
- Eat all the candy in the bowl
Some database operations are naturally idempotent. Consider the “put” operation in a key value store. Setting the value associated with a key is something you can do once or many time and the effect is the same. Some systems
Some operations are much harder. Obviously complex things like long, multi-statement transactions are harder, but even just keeping a counter in state can be really challenging for many systems to guarantee the accuracy of.
Idempotence + At-Least-Once = Exactly Once
Send something more than once, and make sure the effect of processing it is the same no matter how many times it’s processed.
This offers effective exactly-once processing.
How do you make operations idempotent? With ACID transactions, adding idempotency to operations is very easy. SQL INSERTs can be changed to SQL UPSERTs. Adding to a set is usually idempotent. Adding to a list may require an extra query to check for existing membership before it’s added. For example,
INSERT INTO LIST
TEST FOR MEMBERSHIP
IF NOT AN EXISTING MEMBER
INSERT INTO LIST
Strong ACID transactions and idempotency make it easy to preserve exact answers and high throughput in the face of all kinds of failures. Code is saved in fault-handling on the client side; adding idempotency is easier with VoltDB as well.
If you want to know if your application could be made idempotent with VoltDB, reach out to us.