1. Step 1. Download Warp 10™
  2. Step 2. Extract Warp 10™
  3. Step 3. Install Java 8
  4. Step 4. Bootstrap Warp 10™
  5. Step 5. Run Warp 10™
  6. Step 6. Generate Warp 10™ tokens
  7. Step 7. Prepare data in GTS data format
    1. API Endpoint
    2. Data format
    3. Data model example
  8. Step 8. Pushing data
    1. Pushing data with curl
    2. Pushing data with python
  9. Step 8. Quantum interface
  10. Step 9. VSCode IDE for WarpScript™
  11. Next steps

Getting started

This page describes how to install and interact with a standalone version of the Warp 10™ platform. It also describes WarpScript™ IDE tools.

The standalone version of the Warp 10™ platform uses LevelDB as its storage layer, this version is suitable for managing a few tens of millions of time series and few hundreds of billions of datapoints. If you have larger needs, a distributed version of the platform is also available which uses Apache HBase as a scalable storage layer.

Warp 10™ is a server. It uses http requests on different endpoints, one for each task. In this introduction, we will use only two endpoints :

Once you get a Warp 10™ instance running with valid tokens, you do not need physical or ssh access to the server machine anymore to work. Everything could be done with these two endpoints.

In this very first tutorial, you will learn to :

  • start Warp 10™
  • push data
  • read these data in a WarpScript™
  • use Quantum as a WarpScript™ IDE
  • use Visual Studio Code as a WarpScript™ IDE

Step 1. Download Warp 10™

Go to download page, download the latest tar.gz from Bintray.

If you run windows, you can install Warp 10™ with docker. Follow docker instruction, then jump to Step 6.

All commands below are for a Ubuntu test machine. It could be a virtual machine.

Step 2. Extract Warp 10™

Extract the downloaded archive, and go into bin directory.

tar xf warp10-X.Y.Z.tar.gz
cd warp10-X.Y.Z
cd bin

Step 3. Install Java 8

You must have a java 8 JRE installed.

sudo apt update
sudo apt install openjdk-8-jre-headless

The Warp 10™ installer will try to detect automatically your Java path. If you have a custom path or several versions of java installed, you have to define JAVA_HOME explicitly.

Check your default java version:

java -version

If the answer is java 1.8.x_xxx, you can jump to Step 4

If you have several versions of java installed on your system, make sure Warp 10™ will use Java 8. Possible methods are:

  • You can update alternatives on your system with sudo update-alternatives --config java
  • edit bin/warp10-standalone.sh with your favorite text editor, and replace the commented line by your JVM path
  • use command line and adapt the path to your configuration:
sed -i 's/#JAVA_HOME.*/JAVA_HOME=\/usr\/lib\/jvm\/java-8-openjdk-amd64/' warp10-standalone.sh

Step 4. Bootstrap Warp 10™

This will prepare for you the environment of the Warp 10™ platform:

  • Create the warp10 user if needed
  • Generate the standalone configuration file from template
  • Prepare the directory hierarchy
  • Set file permissions

Start bootstrap as root:

sudo ./warp10-standalone.init bootstrap

Then change to warp10 user:

sudo su warp10

While discovering Warp 10™, you will hit some restrictions (number of data in a FETCH, maximum number of operations). All the Warp 10™ configurations are in etc/conf-standalone.conf.

Step 5. Run Warp 10™

As warp10 user:

./warp10-standalone.sh start

You may also launch Warp 10™ as root with sudo ./warp10-standalone.init start

At the first start, this script will create the LevelDB database and all the stuff surrounding it. An initial set of tokens will be provided and some useful commands to start playing with your Warp 10™ instance.

Useful resources:

  • Warp 10™ standalone config conf-standalone.conf has been generated in the etc directory
  • Initials tokens are available in the initial.tokens file in the etc directory
  • Logs are available in the logs directory
  • Data are stored via LevelDB in the leveldb directory

Snapshots of LevelDB data can be performed via the init script. More information here.

Step 6. Generate Warp 10™ tokens

The Warp 10™ platform is built with a robust security model that allows you to have a tight control on who has the right to write and/or read data. The model is structured around the concepts of data producer, data owner, application, within WRITE and READ tokens.

By default, at the first start, Warp 10™ init script will generate both a READ and a WRITE token for the test application io.warp10.bootstrap for a test user that is both the producer and the owner of the data.

Then, you can generate your own write and read tokens by executing the worf command via Warp 10™ init script.

As the warp10 user, generate a token for MyTestApp with a 10 year lifespan :

./warp10-standalone.sh worf MyTestApp 315360000000

Output example :

{
  "read": {
    "token": "hMeJHyhK5wiHur_dsO8kccnzWLAVQO7eOfvOVhXAlmftexalPBy6cENIHwv8JDx10BSBlW0Baz.oOj1uONYYY3H5zdD_WTGePMC_cb2lRr1m_IfUgt.sZYtTdlvU0zbB",
    "tokenIdent": "bce750570d41bb57",
    "ttl": 315360000000,
    "application": "MyTestApp",
    "applications": [
      "MyTestApp"
    ],
    "owners": [
      "1457e6fb-6d6e-4875-a540-2bc25e619b5a"
    ],
    "producer": "1457e6fb-6d6e-4875-a540-2bc25e619b5a",
    "producers": []
  },
  "write": {
    "token": "0RJlPOj4g7SbicLhRq8AAR3egfhMeoJFy7dXm_RtDz5LUGSx3HGlBYvSwa7i_0KIxEqjp9JUPHv4bgBIXJ0TfUyfmvsvQsnD3pNQJ6J3AtoBc77ZiWH.ZF",
    "tokenIdent": "eb8dd24bfe8af277",
    "ttl": 315360000000,
    "application": "MyTestApp",
    "owner": "1457e6fb-6d6e-4875-a540-2bc25e619b5a",
    "producer": "1457e6fb-6d6e-4875-a540-2bc25e619b5a"
  }
}

In order to interact more precisely with the user/token/application system, you may need an interactive access to Warp 10™'s Worf console. More information can be found here.

Step 7. Prepare data in GTS data format

Data is sent into the platform via HTTP POST requests to the Warp 10™ API.

API Endpoint

The HTTP endpoint used to send data is http(s)://host:port/api/vX/update, where vX is the version of the API you want to use (currently v0). In order to be accepted by the platform, requests to this endpoint need to be authenticated, by using a X-Warp10-Token HTTP header with your write token.

Data format

If you have your own data, you can now convert them into GTS format. If you do not have, you can use our example datasets.

  • One GTS is one time series.
  • A GTS is defined by its CLASS and LABELS value.

Knowing this facts, up to your use cases, you must choose wisely what you will put in names and labels.

Data is sent in the body of the POST request, one data point per line. Each line follows the GTS input format:

TS/LAT:LON/ELEV CLASS{LABELS} VALUE

Where:

TS *Optional* Timestamp of the reading, in microseconds since the Unix Epoch. If omitted, the timestamp associated to the data point will be the one of the platform when the point is pushed.
LAT:LON *Optional* geographic coordinates of the reading, using WGS84
ELEV *Optional* elevation of the reading, in millimeters
CLASS Class name of the reading as a URL encoded UTF-8 character string. The encoding of character `{` (Unicode LEFT CURLY BRACKET, 0x007B) is MANDATORY.
LABELS Comma separated list of labels, using the syntax `key=value` where both `key` and `value` are URL encoded UTF-8 character strings. If a key or value contains `,` (Unicode COMMA, 0x002C),`}` (Unicode RIGHT CURLY BRACKET, 0x007D) or `=` (Unicode EQUALS SIGN, 0x003D), those characters MUST be encoded.
VALUE The value of the reading. It can be of one of four types: `LONG`, `DOUBLE` (you must have a dot decimal separator), `BOOLEAN` (character T or F), `STRING` (anything url encoded between single quotes)

Example of valid input lines :

#latitude, longitude, elevation.
959860945800000/37.399386:-3.071806/214748 engine.torque{vehicletype=x21,serial=VF451112448F45B} 206

#no latitude, no longitude, no elevation
1521444669000000// tool_speed{plant=paris,machine=47} 104.392492
=1521444669000020// 105.240093
=1521444669000040// 105.336131
=1521444669000060// 106.168065
#starting a line with "=" means "copy previous names and labels".

#no timestamp, no elevation
/48.0:-4.5/ bar{label0=val0} 3.14

#LONG type
// aa{} -4521
#DOUBLE type
// bb{} -4521.0
#BOOLEAN type
// cc{} T
#STRING type
// dd{} 'it%27s ok'

Data model example

This is a car maker example: Each vehicle records time, position, vehicle speed (km/h), engine rotation speed (rpm), and engine torque (Nm). You plan to look for the time a customer spends on the second gear ratio, and try several models of gear wear, up to torque applied. You can choose to have three different class names:

  • VehicleSpeed
  • EngineRpm
  • EngineTorque

And labels that will help you to identify the GTS:

  • VehicleSerialNumber
  • GearboxType

Because you know the formula will depend on the gearbox, and the gearbox type will never change in the vehicle life. Adding gearbox type as a label will allow you later to select only this gearbox type to perform a computation.

Your resulting GTS text file will look like:

959861036900000/37.399192:-3.071124/214748 VehicleSpeed{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 104.59757588
959861037000000/37.399192:-3.071124/214748 EngineRpm{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 1005.25
959861037000000/37.399192:-3.071124/214748 EngineTorque{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 225
959861046900000/37.399192:-3.071124/214748 VehicleSpeed{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 103.145
959861047000000/37.399192:-3.071124/214748 EngineRpm{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 999.0
959861047000000/37.399192:-3.071124/214748 EngineTorque{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 200

Step 8. Pushing data

In the default configuration, Warp 10™ listens on 127.0.0.1 only.

If you plan to access it from another machine on your network, you must change standalone.host to listen on any interface. Edit config conf-standalone.conf in the etc directory with your favorite text editor, and restart Warp 10™.

Pushing data with curl

If you're on a *nix system, you can push the data using cURL. The Warp 10™ platform listens to the port 8080 of your container. In the setup we have mapped this port to the port 8080 of the host machine, so you can make your request to 127.0.0.1:8080.

POST /api/v0/update HTTP/1.1
Host: host
X-Warp10-Token: wsedrfg.53ef32.detgertyyhefwe5t677456rrfhgbfyhgherdtyreeryer
Content-Type: text/plain

1380475081000000// foo{label0=val0,label1=val1} 123
/48.0:-4.5/ bar{label0=val0} 3.14
1380475081123456/45.0:-0.01/10000000 foobar{label1=val1} T

To test the running container, push a single GTS containing one data in the platform using your WRITE token.

As any user, or maybe on another machine.

curl -v -H 'X-Warp10-Token: 0RJlPOj4g7SbicLhRq8AAR3egfhMeoJFy7dXm_RtDz5LUGSx3HGlBYvSwa7i_0KIxEqjp9JUPHv4bgBIXJ0TfUyfmvsvQsnD3pNQJ6J3AtoBc77ZiWH.ZF' --data-binary "1// test{} 42" 'http://127.0.0.1:8080/api/v0/update'

If everything is OK, you should receive a HTTP 200

Create a text file with your favorite editor, with more lines, save it as "example.gts":

959861036900000/37.399192:-3.071124/214748 VehicleSpeed{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 104.59757588
959861037000000/37.399192:-3.071124/214748 EngineRpm{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 1005.25
959861037000000/37.399192:-3.071124/214748 EngineTorque{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 225
959861046900000/37.399192:-3.071124/214748 VehicleSpeed{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 103.145
959861047000000/37.399192:-3.071124/214748 EngineRpm{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 999.0
959861047000000/37.399192:-3.071124/214748 EngineTorque{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 200

Then push the file with curl (chunked is useful to push huge files that does not fit in RAM):

curl -v  -H 'Transfer-Encoding: chunked' -H 'X-Warp10-Token: 0RJlPOj4g7SbicLhRq8AAR3egfhMeoJFy7dXm_RtDz5LUGSx3HGlBYvSwa7i_0KIxEqjp9JUPHv4bgBIXJ0TfUyfmvsvQsnD3pNQJ6J3AtoBc77ZiWH.ZF' -T example.gts 'http://127.0.0.1:8080/api/v0/update'

Get all these data using your READ token.

curl -v  --data-binary "[ 'hMeJHyhK5wiHur_dsO8kccnzWLAVQO7eOfvOVhXAlmftexalPBy6cENIHwv8JDx10BSBlW0Baz.oOj1uONYYY3H5zdD_WTGePMC_cb2lRr1m_IfUgt.sZYtTdlvU0zbB' '~.*' {} NOW -1 ] FETCH" 'http://127.0.0.1:8080/api/v0/exec'

If everything is OK, you should receive a HTTP 200 OK with the most recent datapoint of every GTS in JSON format.

When using Docker on Mac OS or Windows, there is no binding between Warp 10™ API address and the host (docker is ran throw a Virtual Machine). To reach Warp 10™ you need to replace 127.0.0.1 by the real IP address of the container. To get it, use a simple docker-machine ip default>, the container address is also shown in the Settings/Ports page of your container. If you used the shared volume between the container and the host, you can access to the virtual machine using docker-machine ssh default> and inspect the repertory /var/warp10. Don't hesitate to check on docker-machine documentation.

Pushing data with python

This small python program will take GTS text file as arguments, and will push them using requests library.

#!/usr/bin/env python
#  -*- coding: utf-8 -*-
import sys
# sudo apt install python-pip
# sudo pip install requests
import requests

if len(sys.argv) < 2:
    print ("Use : " + sys.argv[0] + " *.gtstxt")
    exit()

for filename in sys.argv[1:]:
    print("opening " + filename)
    fout = open(filename, 'rb')
    GTSfile = fout.read()
    fout.close()
    headers = {
        "X-Warp10-Token": "0RJlPOj4g7SbicLhRq8AAR3egfhMeoJFy7dXm_RtDz5LUGSx3HGlBYvSwa7i_0KIxEqjp9JUPHv4bgBIXJ0TfUyfmvsvQsnD3pNQJ6J3AtoBc77ZiWH.ZF"}
    url = "http://127.0.0.1:8080/api/v0/update"
    print(" push data to warp10... " + url)
    r = requests.post(url, headers=headers, data=GTSfile)
    print(" warp10 server answer " + str(r.status_code))

Step 8. Quantum interface

Warp 10™ can be extended with plugins and extensions. Quantum is a plugin delivered and preconfigured with the standalone or docker version. Warp 10™'s Quantum is a web application aiming to allow users to interact with the platform in an user-friendly way, offering an alternative to command-line interaction.

Open http://localhost:8090/ in your favorite browser (if needed, replace localhost by the IP of the Warp 10™ host).

By default, Quantum API target is the historical Warp 10™ distributed instance https://warp.senx.io/api/v0. You need to replace the API with your Warp 10™ machine. Click choose another backend, type your machine API URL, and click on the validation button:

Quantum - Configuration

Once backend configuration is done, you can execute WarpScript on your Warp 10™ instance. For example, the very first WarpScript you type in this tutorial, using curl, to read all data you pushed:

//store read token in a variable
'hMeJHyhK5wiHur_dsO8kccnzWLAVQO7eOfvOVhXAlmftexalPBy6cENIHwv8JDx10BSBlW0Baz.oOj1uONYYY3H5zdD_WTGePMC_cb2lRr1m_IfUgt.sZYtTdlvU0zbB' 'myreadtoken' STORE

//fetch the most recent point of every GTS this token can access
[ $myreadtoken '~.*' {} NOW -1 ] FETCH

Quantum - Configuration

Step 9. VSCode IDE for WarpScript™

Visual Studio Code is a source code editor developed by Microsoft for Windows, Linux and macOS. It includes support for debugging, embedded Git control, syntax highlighting, intelligent code completion, snippets, and code refactoring. It is free and open source. We developed a plugin for WarpScript files (.mc2 extension).

After installing VS Code, launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.

ext install senx.warpscript-language

Click install and restart VS Code.

Create a test.mc2 file with your warpscript:

//store read token in a variable
'hMeJHyhK5wiHur_dsO8kccnzWLAVQO7eOfvOVhXAlmftexalPBy6cENIHwv8JDx10BSBlW0Baz.oOj1uONYYY3H5zdD_WTGePMC_cb2lRr1m_IfUgt.sZYtTdlvU0zbB' 'myreadtoken' STORE

//fetch the most recent point of every GTS this token can access
[ $myreadtoken '~.*' {} NOW -1 ] FETCH

If the extension is correctly loaded, you now have syntax highlighting, autocompletion, hover help, variable completion, local macro substitution...

VSCode - first time

By default, VSCode extension will execute WarpScript on localhost API (http://127.0.0.1:8080/api/v0/exec). To change the Warp 10™ server address, open the VSCode preferences (Ctrl+Shift+P, type "settings", press enter), search for "warpscript", and copy paste a customized line from the left pane to the right pane. If you plan to use multiple instances of Warp 10™, you can also specify an endpoint in the very first comment line of the file. VSCode will interpret it.

Press Ctrl+Alt+E to run the warpscript.

VSCode - first time

JSON result will be displayed on the right part of the editor. In this example, we count the number of GTS returned by the FETCH command. Up to context, JSON can context GTS or PNG images. WarpScript extension will also open GTS Preview or Image Preview tabs.

For advanced use, read our VSCode help page.

Next steps

Congratulation, you now have a fully working environment to evaluate Warp 10™ and learn WarpScript™.

You now need to learn WarpScript™ syntax and discover the functions. First of all, read the WarpScript concepts. Then you can follow the tutorials or you can search for functions and examples in the reference documentation. We do provide datasets in the tutorials but the best way is to learn with your data and your use cases!