To Go Fast – Move Code to Data
Data is almost always bigger and more frequently changing than logic, yet so many platforms ask users to move data to the logic. Flip that around and you can substantially reduce both network traffic and end-to-end latency of complex processing.
One procedure. One transaction. One network round trip.
Successful projects running complex transactions at scale already use stored procedures to go fast. VoltDB puts stored logic at the forefront of its transaction model and reaps the performance rewards of that choice. But stored procedures have detractors. How are VoltDB’s procedures different that the stored procedures in existing relational databases?
- Use a programming language you know or can learn easily (Java).
- Debug server-side code the same way you debug local code, on the CLI or in an IDE.
- Include third-party libraries, helper code, or other resources.
- Test procedures with standard unit testing frameworks.
- You can even use printf-style debugging.
The VoltDB stored procedure model has additional benefits over client-side applications. In some cases, you can build abstractions between the client and the state, a “Data API” so to speak. For an example, see our Unique Devices example (GitHub).
Natural Event Processing
VoltDB stored procedures are a great fit for event and stream processing. An event arrives; stored logic is ready to handle it. Unlike systems such as Storm and Spark Streaming, VoltDB is able to respond to the caller directly, transactionally query and mutate state, and push zero or more tuples to a downstream system, e.g. an OLAP store or HDFS.
Use some of this functionality or all of it; VoltDB’s flexibility, integration and strong consistency is a refreshing option in a sea of fragile or limited choices.
VoltDB offers a number of SQL Functions built-in, but sometimes it would be helpful to have custom logic inside a SQL statement. User-Defined functions (UDFs) allow you load custom code like stored procedures, but use them inside queries.
- Add helpful conversion functions.
- Add an estimator or probabilistic function.
- Parse binary formats and decompress things.
UDFs can make exiting applications simpler, or they can make new applications possible. A number of customers are beginning to use UDFs in their fraud detection systems for cleaner application of machine learning models in real time.
Importers and Exporters
VoltDB has a whole page about ingest (importers) and export, but this is also another area where users can load code into VoltDB to make their system more powerful.
When a message comes in from a Kafka topic, or an AWS Kinesis stream, it might not be perfectly ready to pass into a procedure. Custom import formatters can convert special formats, filter or pre-process messages on their way to VoltDB.
Export adapters are a bit more serious, but if you want to connect a VoltDB export stream to a new target, VoltDB has a framework for that to ensure at-least-once delivery in the face of all kinds of problems.