Enterprise Stream Processing - VoltDB
page-template-default,page,page-id-14871,page-child,parent-pageid-10675,mkd-core-1.0,highrise-ver-1.0,,mkd-smooth-page-transitions,mkd-ajax,mkd-grid-1300,mkd-blog-installed,mkd-header-standard,mkd-sticky-header-on-scroll-up,mkd-default-mobile-header,mkd-sticky-up-mobile-header,mkd-dropdown-slide-from-bottom,mkd-dark-header,mkd-header-style-on-scroll,mkd-full-width-wide-menu,mkd-header-standard-in-grid-shadow-disable,mkd-search-dropdown,wpb-js-composer js-comp-ver-6.1,vc_responsive
VoltDB / Product  / Database Features & Benefits / Enterprise Stream Processing

Enterprise Stream Processing

Apache Kafka & Amazon Kinesis are terrific tools for building real-time data streaming pipelines, but performing transactional processing and analytics on stand-alone message bus technology can be challenging. With a combination of real-time actionable data processing capabilities such as: contextual state, ACID transactions, and embedded machine learning along with fast ingestion and export; you can drastically reduce: complexity, latency, and hardware costs. Enabling use cases such as smart meter billing, predictive maintenance in Industrial IoT, fraud prevention, personalized offers, and more.

Traditional databases are fast, but lack streaming capabilities. While traditional streaming architectures like Kappa can do fast streaming, but offer only very rudimentary data management. VoltDB’s Smart Streaming architecture significantly simplifies the Kappa Architecture. In the Kappa architecture; Apache Kafka is used as a real-time glue for ingestion because of the need for durability and speed mismatches downstream. VoltDB is fast enough to ingest and offers durability built-in, eliminating the need for an Apache Kafka ingestion layer. In VoltDB, the data is stored in a relational format therefore a separate serving database isn’t required either. Finally, VoltDB has complex event processing capabilities like aggregations, filtering, sampling, correlations, along with stored procedures and user defined functions. Machine Learning models can be embedded within Java based stored procedures to provide real-time decisions on fast streaming data. Smart Streaming brings analysis to the data as opposed to taking the data to the analysis, resulting in:

  • Higher Performance & Scale
    The Smart Streaming architecture can handle terabytes of stateful data coupled with the ability to stream and operationalize millions of events in milliseconds.
  • Faster Analysis and Decision Making
    VoltDB can queue / enrich data, perform rich analysis, and operationalize complex ML models in one integrated environment. For streaming processing, VoltDB features:

    • Data enrichment and transformation features like aggregates & windows
    • The strongest ACID guarantee in the industry.
    • Full ANSI SQL support.
    • Automatic data migration to downstream systems.
    • Embedded ML in live streams to take smart decisions in real-time.
  • Easier Operations
    These requirements include requirements needed to deploy Smart Streaming in mission critical applications

    • Stream management functions like pause, resume and skip records in case of failure
    • Ability to publish and subscribe to different data streams
    • Durability features like Command Log/Snapshots to persist a power loss or failure.
    • High availability features that can tolerate node failures.
    • Replication features for disaster recovery and having a distributed mesh of clusters with active/active replication between them.
    • Detailed monitoring features that allow statistics to be collected to make sure everything is up and running.
Stream Processing with VoltDB
Stream Processing Ingestion

You can seamlessly bulk-load data into VoltDB (tables or calling procedures) from Apache Kafka, using any of the available connectors:

  • Apache Kafka Loader
    Is a CLI tool that runs as a local process, configure each process for one topic and one table or procedure. Run multiple processes for multiple topics.
  • Apache Kafka Importer
    Runs in threads as part of the VoltDB database process, distributed with built-in redundancy. It can be configured for multiple topics, tables, and procedures.
  • Apache Kafka Connect Sink Connector
    Runs within Kafka Connect. It can configure for multiple topics, tables, and procedures.

For Amazon Web Services Users

Learn more about fast data ingest and export.

Stream Processing

The Stream Processing heart of the Smart Streaming architecture was designed to deliver business value in real-time for use cases that need: complex actionable decisions with state, and data accuracy / consistency.

Unlike existing solutions; Smart Stream Processing utilizes VoltDB’s proven in-memory operational database that provides contextual state, strict ACID semantics, and embedded Machine Learning (“smart”) without compromising on scale, performance, availability, fault tolerance, and low latency. These capabilities are essential to the success of use cases, such as: Smart meter billing, predictive maintenance in Industrial IoT, fraud prevention, real-time personalization/offers, and more.

Stream Processing Export

VoltDB lets you export data to a Kafka stream out-of-the-box with a variety of options to match your individual data pattern:

  • TTL Migration
    Allows you to automatically migrate data to either a stream or a data lake. VoltDB maintains a copy of the migrating data in-database, until it receives confirmation that the data is loaded either in the data lake or data warehouse.
  • Apache Kafka Export Connector
    Runs in threads as part of the VoltDB database process. Each stream can be pushed to a topic.
  • Amazon Kinesis Firehose Export Conduit
    The Amazon Kinesis streams and firehose export conduit enables you to easily export data from VoltDB to Amazon Kinesis without any application coding.

Learn more about fast data ingest and export.