VoltDB – Creating an Openshift Cartridge – Part 2

Thursday, April 3, 2014

Originally published by Jeff McCormick on March 31, 2014. See part 1 here.


In this blog post, I’ll show you the steps required to fix a couple of things in our original cartridge.

Determine the Openshift Internally Assigned IP Address

First, lets make our cartridge capable of being installed onto any Openshift web framework application, not just the DIY cartridge.


In Openshift, an internal IP address is assigned to your application when the web framework is used to create the initial application. For example, with the DIY cartridge, the internally assigned IP address is found in the environment variable called OPENSHIFT_DIY_IP. And if you were to select the Python web framework, the environment variable is called OPENSHIFT_PYTHON_IP, and for the PHP framework the variable is called OPENSHIFT_PHP_IP, and so on.


The naming convention would tend to cause you to write scripts, like the ones for our VoltDB cartridge as follows:


internalAddress = OPENSHIFT_DIY_IP
else if OPENSHIFT_PHP_IP thne
internalAddress = OPENSHIFT_PHP_IP
internalAddress = OPENSHIFT_PYTHON_IP
else if…….


This is however not required given that within the OPENSHIFT_BASH_SDK script there exists two useful bash functions named:




These two functions can be used to determine the underlying web framework’s IP address as follows:


function getinternalip() {
echo $ipaddressvar
echo ${!ipaddressvar}


Including this function into the VoltDB’s bin/control script now allows us to start and stop VoltDB using a generic internal IP address as follows:


function start() {

nohup voltdb create –internal 16700 –externalinterface $internaladdressip –zookeeper 16300 –httphost $internaladdressip -d $OPENSHIFT_VOLTDB_DIR/sample/deployment.xml –zkhost $internaladdressip -H $internaladdressip –internalinterface $internaladdressip $OPENSHIFT_VOLTDB_DIR/sample/voltcache.jar > $OPENSHIFT_VOLTDB_DIR/logs/startup.log &

function stop() {
client_result “stopping voltdb….”
client_result “voltdb is running …issuing stop “;
export HOME=${HOME}/app-root/data
voltadmin pause -H $internaladdressip;
voltadmin shutdown -H $internaladdressip;
return 1


You now see that our control script is agnostic as to what web framework we are installing VoltDB on.

Deploying Our Own Catalog

Next, we would like to alter the cartridge to allow us to run any VoltDB schema of our choosing. This is accomplished simply by changing the scripts to use a generic application schema name and deployment file name in the start() function as follows:


nohup voltdb create –internal 16700 –externalinterface $internaladdressip –zookeeper 16300 –httphost $internaladdressip -d $OPENSHIFT_VOLTDB_DIR/sample/currentdeployment.xml –zkhost $internaladdressip -H $internaladdressip –internalinterface $internaladdressip $OPENSHIFT_VOLTDB_DIR/sample/currentschema.jar > $OPENSHIFT_VOLTDB_DIR/logs/startup.log &


Instead of running the sample catalog for voltcache everytime, we will adjust our cartridge scripts to allow us to run a catalog of your choosing.


In the cartridge control script, we will modify the paths to the catalogs being executed as follows:


nohup voltdb create –internal 16700 –externalinterface $internaladdressip –zookeeper 16300 –httphost $internaladdressip -d $OPENSHIFT_VOLTDB_DIR/sample/currentdeployment.xml –zkhost $internaladdressip -H $internaladdressip –internalinterface $internaladdressip $OPENSHIFT_VOLTDB_DIR/sample/currentschema.jar > $OPENSHIFT_VOLTDB_DIR/logs/startup.log &


Notice that we make use of the OPENSHIFT_DATA_DIR environment variable. This location is writable by your account and is more generic than the sample directory which gets deployed with the cartridge itself. Now, the cartridge will look in this location for a catalog to start.


Initially when the cartridge is installed, the generic name will point to the voltcache example to provide a default example to install the cartridge with. This is performed by means of the cartridge’s setup script as follows:


cp ~/voltdb/sample/voltcache.jar $OPENSHIFT_DATA_DIR/currentschema.jar
cp ~/voltdb/sample/deployment.xml $OPENSHIFT_DATA_DIR/currentdeployment.xml


To supply your own schema and deployment files is simple if we make use of the rhc client utilities. In this example, we’ll upload the voltkv example that ships with VoltDB. Make sure to build the voltkv example on your local workstation before running these commands! In the following example, the application name we created is called voltdb. From your client workstation (e.g. Fedora 20), you will enter the following commands to upload your own schema and deployment files:


$ cd ./voltdb-
$ rhc cartridge stop jeffmccormick-voltdb- -a voltdb
Stopping jeffmccormick-voltdb- …
Failed to execute: ‘control stop’ for
$ rhc cartridge status jeffmccormick-voltdb- -a voltdb
voltdb is not running
$ rhc scp voltdb upload deployment.xml app-root/data/currentdeployment.xml
$ rhc scp voltdb upload voltkv.jar app-root/data/currentschema.jar
$ rhc cartridge start jeffmccormick-voltdb- -a voltdb
Starting jeffmccormick-voltdb- … done
starting voltdb….
voltdb is not running…starting it now


Remember to add the http port setting of 16000 (port=”16000″) in your deployment.xml files or VoltDB will not start on Openshift!


Now, on your client workstation, start port forwarding in one terminal window:


$ rhc port-forward -a voltdb


Then in another terminal window, run the voltkv client:


./run.sh client
jeffmc/edisk/tools/voltdb-]: ./run.sh client
Command Line Configuration
displayinterval = 5
duration = 30
entropy = 127
getputratio = 0.9
keysize = 32
latencyreport = false
maxvaluesize = 1024
minvaluesize = 1024
poolsize = 1000
preload = true
ratelimit = 2147483647
servers = localhost
statsfile =
usecompression = false
warmup = 5
Setup & Initialization
Connecting to VoltDB…
Connected to VoltDB node at: localhost.

Preloading data store…
Preloading complete.
Starting Benchmark
Warming up…

Running benchmark…
00:00:05 Throughput 640/s, Aborts/Failures 0/0
00:00:10 Throughput 816/s, Aborts/Failures 0/0
00:00:15 Throughput 621/s, Aborts/Failures 0/0
00:00:20 Throughput 998/s, Aborts/Failures 0/0
00:00:25 Throughput 1155/s, Aborts/Failures 0/0
00:00:30 Throughput 1110/s, Aborts/Failures 0/0
KV Store Results
A total of 29,717 operations were posted…
– GETs: 24,020 Operations (0 Misses and 0 Failures)
23 MB in compressed store data
23 MB in uncompressed application data
Network Throughput: 0.006 Gbps*
– PUTs: 2,707 Operations (0 Failures)
2 MB in compressed store data
2 MB in uncompressed application data
Network Throughput: 0.006 Gbps*
– Total Network Throughput: 0.012 Gbps*
* Figure includes key & value traffic but not database protocol overhead.
Client Workload Statistics


Average throughput: 921 txns/sec


After this set of changes to the cartridge, you can now run VoltDB on any of the supported Openshift web frameworks, and also deploy your own VoltDB catalog.


Whats next? Well, VoltDB has many features that relate to high availability and multi-node configuration, those are enterprise style features that are particularly challenging to implement and I’ll address those in upcoming blogs.

by VoltDB