1. Use case
  2. REXEC introduction
  3. Serialization
  4. Remote update
  5. Pushing data on the datalake
    1. Enhance performances:
    2. Enhance security:

Warp 10™ talking to Warp 10™

In this tutorial, you will learn several ways to interact between different Warp 10™ instances. You do not need any external tooling to do this. You need two Warp 10™ instances on your local network.

Use case

An embedded datalogger collect data on CAN, ModBus, analog inputs. It stores data locally in a local Warp 10™ instance. Each minutes, it performs a subsampling of the data and push them in the company Warp 10™ datalake.

In this tutorial, my embedded datalogger IP is 192.168.1.105, and my company datalake is my own computer 192.168.1.210.

On both Warp 10™ instances, I have read and write tokens, named below:

  • DataLakeR: read token on 192.168.1.210
  • DataLakeW: read token on 192.168.1.210
  • LoggerR: read token on 192.168.1.105
  • LoggerW: write token on 192.168.1.105

Note you can really define a token called LoggerR. Not really secure, but very handy for a tutorial, you can copy paste examples without any replacement. Read more here.

REXEC introduction

You build a custom application to listen on CAN, ModBus, analog inputs. This application produces GTS input format, send to the local ingress endpoint http://localhost:8080/api/v0/update. Now, you need to copy these data on your company datalake.

WarpScript has a REXEC function which allow any WarpScript to execute WarpScript on another Warp 10™ instance. This function, with a few serialization tricks, is the key to easily transfer anything between two Warp 10™ instances.

For this tutorial, you need to allow REXEC on the datalogger Warp 10™. In your configuration file set property warpscript.rexec.enable to true. Restart the instance and test it:

RexecTest.mc2 file content:

// @endpoint http://192.168.1.105:8080/api/v0/exec '2 2 +' 'http://localhost:8080/api/v0/exec' REXEC

Note the trick at the first line of my WarpScript file... I am using VSCode IDE with WarpScript language extension, and I dynamically ask VSCode to change the execution endpoint. See VSCode guide here.

The result is a stack with the number 4. What happened ?

  • The top of the stack is the execution endpoint (API url).
  • Just below, I put a string. This string is a WarpScript. REXEC took the WarpScript and exec it on itself (localhost).

Note: The EVAL function does exactly the same, locally. It parses and executes WarpScript in a string. A WarpScript is able to generate WarpScript and execute it.

'2 2 +' EVAL

I can ask my embedded Warp 10™ to execute WarpScript on my company datalake. First, make sure your Warp 10™ listens on every interfaces, not only 127.0.0.1 (default setting in a standalone version). Edit the configuration file, and set standalone.host = 0.0.0.0. Restart and test it:

// change the VSCode execution endpoint to the datalogger IP // @endpoint http://192.168.1.105:8080/api/v0/exec // execute a warpscript on the company datalake '2 3 +' 'http://192.168.1.210:8080/api/v0/exec' REXEC

The result is 5. The datalogger executed 2 3 + on the company datalake.

In the getting started tutorial, you learned to push data to the ingress endpoint /api/v0/update. The great news is that you can push data into the storage from within a WarpScript with the UPDATE function:

// change the VSCode execution endpoint to the datalogger IP // @endpoint http://192.168.1.105:8080/api/v0/exec // generate some data inside the datalogger, write them locally NEWGTS 'externaltemperature' RENAME 'mygts' STORE //rename and store NOW NOW 10 m + <% 1 s + %> //from now to now + 10 minutes with a 1 s step <% 'timestamp' STORE // the computed index of the for loop is my timestamp RAND 10.0 * 15.0 + 'temperature' STORE //random temperature between 15 and 25 °C $mygts $timestamp NaN NaN NaN $temperature ADDVALUE DROP %> FORSTEP // 601 data points //store them in logger Warp10 $mygts 'LoggerW' UPDATE

If you replaced LoggerW by the correct write token, you should have an empty stack result without any error, and you can read the data back from storage with the matching read token:

// @endpoint http://192.168.1.105:8080/api/v0/exec // read data from datalogger storage [ 'LoggerR' 'externaltemperature' {} NOW -1000 ] FETCH //get latest 1000 data points

Serialization

You need to build a string which is a valid WarpScript, and REXEC it on your datalake with an UPDATE instruction. Here enter serialization: WarpScript offers off the shelf functions to serialize easily your data. If you didn't, it is time to go back in the learning path and read the last part of Type conversion tutorial.

Serialization is your best friend to exchange data between Warp 10™ instances. First, test it:

// @endpoint http://192.168.1.105:8080/api/v0/exec // read data from datalogger storage [ 'LoggerR' 'externaltemperature' {} NOW -1000 ] FETCH //get latest 1000 data points WRAP //wrap the result

Input is a list of GTS, output is a list of strings. Each GTS has been individually serialized. To serialize a list of string, you can turn it in a JSON.

Remote update

// @endpoint http://192.168.1.105:8080/api/v0/exec 'DataLakeW' 'LakeWriteToken' STORE //store the token to write on the remote server. // read data from datalogger storage [ 'LoggerR' 'externaltemperature' {} NOW -1000 ] FETCH //get latest 1000 data points [ SWAP bucketizer.mean 0 10 s 0 ] BUCKETIZE //subsample to 10 s period, keeping the mean of each bucket WRAP //wrap the result ->JSON //serialize the list 'wrappeddata' STORE //store the result //build a warpscript '"' //you can use either simple quotes or double quotes to build a string in WarpScript $wrappeddata + //concatenate '" JSON-> "' + $LakeWriteToken + '" UPDATE' + 'mywarpscript' STORE //run the warpscript on the datalake server. $mywarpscript 'http://192.168.1.210:8080/api/v0/exec' REXEC

The above example seems correct. But it fails with error 500.

Explanations:

  • Erasing your own data with UPDATE is easy. For example, you FETCH some data, apply a mean mapper, then UPDATE the result: oops, you have just erased your original data. Warp 10™ has a security mechanism to prevent this: if you do not rename a GTS, you cannot UPDATE it. The 'renamed' flag is also wrapped by the serialization process.
  • You FETCH some GTS on the datalogger, but you never renamed them. UPLOAD is forbidden. You need to rename them, before or after serialization. Renaming with their original name is allowed.

In the following example, the rename is done on the server side:

// @endpoint http://192.168.1.105:8080/api/v0/exec 'DataLakeW' 'LakeWriteToken' STORE //store the token to write on the remote server. // read data from datalogger storage [ 'LoggerR' 'externaltemperature' {} NOW -1000 ] FETCH //get latest 1000 data points [ SWAP bucketizer.mean 0 10 s 0 ] BUCKETIZE //subsample to 10 s period, keeping the mean of each bucket WRAP //wrap the result ->JSON //serialize the list 'wrappeddata' STORE //store the result //build a warpscript '"' //you can use either simple quotes or double quotes to build a string in WarpScript $wrappeddata + //concatenate '" JSON-> UNWRAP <%25 DROP DUP NAME RENAME %25> LMAP "' + $LakeWriteToken + '" UPDATE' + 'mywarpscript' STORE //run the warpscript on the datalake server. $mywarpscript 'http://192.168.1.210:8080/api/v0/exec' REXEC
  • LMAP is used to apply a macro on each list elements.
  • Remember, WarpScript strings are URL encoded. You need to escape % character with %25.

You can check if the data is on your datalake:

// execute on the datalake // @endpoint http://192.168.1.210:8080/api/v0/exec // read data from datalogger storage [ 'DataLakeR' 'externaltemperature' {} NOW -1000 ] FETCH //get latest 1000 data points

SNAPSHOT the stack

You may want to serialize other elements than GTS lists. SNAPSHOT can serialize the full stack content in a valid warpscript string. Note that SNAPSHOT serializes GTS with WRAP. The SNAPSHOT output is ready to EVAL (or REXEC).

// @endpoint http://192.168.1.105:8080/api/v0/exec [ NEWGTS 'a' RENAME NEWGTS 'b' RENAME ] // a list of GTS { 'action' 'RawSensorAnalyse' } // a map <% DROP %> // a macro ( 4 5 6 ) //a SET (cannot be represented on the stack, you see null) SNAPSHOT 'serializedWarpScript' STORE //all the stack content is serialized $serializedWarpScript // push it on the stack //execute it remotely. $serializedWarpScript 'http://192.168.1.210:8080/api/v0/exec' REXEC //SET-> //display the SET

Pushing data on the datalake

Now you know how to transfer easily GTS from On the embedded Warp 10™ instance, you need to create a runner every minutes.

To keep track of successful uploads, you can create a GTS 'successfullupload' on the datalogger. This GTS will contains the timestamp of the last successful update.

Enhance performances:

  • Use REXECZ. In both ways, the content will be gzipped on the fly, saving bandwidth.

Enhance security:

  • Use https with good certificates.
  • Use tokens. Tokens allow you to do a pretty cool setup in this case: Every datalogger can have access to read and write data but not delete them. If a datalogger is compromised, the attacker won't be able to access more than this datalogger data.