Top
A Downloadable Demo of VoltDB New TASK and MIGRATE Features - VoltDB
16262
post-template-default,single,single-post,postid-16262,single-format-standard,mkd-core-1.0,highrise-ver-1.0,,mkd-smooth-page-transitions,mkd-ajax,mkd-grid-1300,mkd-blog-installed,mkd-header-standard,mkd-sticky-header-on-scroll-up,mkd-default-mobile-header,mkd-sticky-up-mobile-header,mkd-dropdown-slide-from-bottom,mkd-dark-header,mkd-header-style-on-scroll,mkd-full-width-wide-menu,mkd-header-standard-in-grid-shadow-disable,mkd-search-dropdown,wpb-js-composer js-comp-ver-6.0.5,vc_responsive
VoltDB / Announcements  / A Downloadable Demo of VoltDB New TASK and MIGRATE Features

Blog

A Downloadable Demo of VoltDB New TASK and MIGRATE Features

Third post in a series on features of VoltDB Version 9.2.

Version 9.2 of VoltDB introduces two new features that are highly relevant to people developing streaming applications. This is part three of a series of three blog posts:

Introducing MIGRATE
Introducing CREATE TASK

This demo showcases some VoltDB features:

  • The MIGRATE command.
  • Directed Procedures and Scheduled Tasks.

All of the code is available here.

Scenario

We are pretending to be a system for detecting rogue drones in central London. Various locations are forbidden to drones. We need to raise the alarm if a drone gets too close to one of these locations. We also need to raise an alarm if we stop getting reports for a drone. We are expecting 10’s of thousands of position reports per second.

Schema

We have the following tables, views and export streams:

NameTypePurpose
DronesTableMaster table for drones. Used for tracking when to declare it missing.
Important_LocationsTablePlaces in Central London we want to keep an eye on
drone_locationsTableKeeps last 10 position reports for each drone. Extra rows are MIGRATED to old_drone_locations_tgt
missing_dronesExport StreamWe add a record every time a drone stops sending us location information
location_incursionsExport StreamWe add a record every time a drone gets too close to a location mentioned in Important_locations
old_drone_locations_tgtExport StreamWhere old drone_location records go. Defined in the DDL for DRONE_LOCATIONS
drone_activityViewShows how many drones have reported positions over the last few minutes
missing_drone_statsViewShows how many drones will be declared missing over the next few minutes if we don’t get a location report
missing_drone_maxdateViewA single row that tells us whether we need to search for missing drones
latest_drone_activityViewA summary showing the latest location report time for each drone
Procedures

The application has two stored procedures:

ReportLocation

‘ReportLocation’ is fed mock data by the test client (TaskMigrateDemoClient), but in a real world deployment would be configured to read messages from a queue such as Kafka or Kinesis.

ReportLocation takes a position report for a drone and updates the database. It also:

  • Checks to see if the drone is too close to an important_location.
  • MIGRATES any extra drone_location records

This is called repeatedly from the demo’s client program.

FindStaleDroneReports

FindStaleDroneReports finds any drones that have failed to report for too long a time period and writes a message to missing_drone_stats. It also updates the drone record to prevent duplicate reports. FindStaleDroneReports is run as a TASK every 250 milliseconds

Installation and setup

To run this you need to go through the following steps:

  • Setup VoltDB. It can be downloaded from here.
  • Get the source for this demo from here.
  • The source has a dependency on a utility for creating DDL objects, which is available here.
  • The VoltDB classes voltdb-9.2.jar and voltdbclient-9.2.jar (or later) will need to be in your classpath
  • Compile and run the class TaskMigrateDemoClient.java.
  • The first time you run the client (see below) it will create all the database objects for you.
  • Once you have the application running you can then configure export streams (see below)

Configure Export Streams

In order to fully observe the functionality of this demo you’ll need to get export up and running. This can be done in two ways:

Configure Streams from within the VoltDB GUI

This can be done by going to the Admin tab of the VoltDB Management Console GUI and creating entries like these:

Migrate and Create Task

Note that you’ll need to change ‘outdir’ to a directory that exists on your computer. You could also set this up to go to Kafka (for example) by selecting it from the ‘type’ dropdown.

Configure streams by shutting down the database and editing deployment.xml

An alternative approach is to shut down VoltDB and then add the following entries to the database’s deployment.xml configuration file:

csv odlt /Users/drolfe/csv csv md /Users/drolfe/csv csv lit /Users/drolfe/csv

When you restart the database the export configuration should look like the one above.

Run the Code

The client program is called TaskMigrateDemoClient and takes the following parameters:

ParameterPurposeExample
hostnamescomma delimited list of VoltDB hosts127.0.0.1
tpsHow many transactions per second to generate50000
sizeHow many drones to track1000000
secondsHow many seconds to run for180

TaskMigrateDemoClient 192.168.0.50,192.168.0.51 50000 10000000 1800

Note that you may have to try various combinations of tps and size to get interesting results. The higher tps is the more records get written to ‘drone_locations’, and the higher the value of ‘size’ the more likely it is that locations will get old enough to be flagged as ‘missing’ and reported to ‘missing_drones’.

When running it calls the procedure GetStatus every 100,000 iterations, so you can see aggregate information about what is going on.

The first time it runs it starts by creating all the objects needed:

It then sends ReportLocation messages to the server, at a rate of ‘tps’ per second. Every 100,000 events we call GetStatus, which show how many drones have been active over the last few minutes and how many appear to be missing:

2019-12-16 08:54:37:Starting test run at 30000 transactions per second for 1800 seconds

2019-12-16 08:54:41:Transaction #100000

2019-12-16 08:54:41:Drone Activity By Minute:

2019-12-16 08:54:41:

LAST_ACTIVE             HOW_MANY

————————— ———

2019-12-16 08:54:00.000000 100000

 

2019-12-16 08:54:41:Missing Drones By Minute:

2019-12-16 08:54:41:

HOW_MANY

———

36225

 

2019-12-16 08:54:45:Transaction #200000

2019-12-16 08:54:45:Drone Activity By Minute:

2019-12-16 08:54:45:

LAST_ACTIVE             HOW_MANY

————————— ———

2019-12-16 08:54:00.000000 200000

 

2019-12-16 08:54:45:Missing Drones By Minute:

2019-12-16 08:54:45:

HOW_MANY

———

69825

When running the ‘Db Monitor’ tab of the VoltDB GUI should look like this:

Db Monitor tab VoltDB GUI

As the demo progresses the Export tab will show records being written:

Export Connections VoltDB

Meanwhile if you look at the directory you requested the file exporter to write to you’ll see files being written:

 

Conclusion

In this demo we’ve shown how you can use VoltDB to create a fast, scalable application that takes important, stateful decisions in real time. In addition to showing how we can correlate the latest position of a device with its proximity to a location we also show how we can make smart decisions, such as reporting a device as missing once (and only once).