Loading

Connecting industrial equipment to Warp 10

As an industrial equipment supplier, your customer asked you to be compatible with the Warp 10 Platform. This page is here to help you to fulfill this requirement, for devices as small as 100 kB RAM microcontrollers (without RTC), up to high-end acquisition systems.

intro

Warp 10 is tailored for industrial time series.

It is designed to efficiently store sensor data, events, status, without the drawbacks of SQL or SQL-based time series databases. You don't need to store regular timestamps, you don't need to align timestamps, you can write far in the past to catch up data history after a long offline period.

Warp 10 is designed to ingest chunks of data (one request per datapoint has an overhead cost). Depending on your customer requirements, chunk's length can be 10 seconds or several hours or days.

The Warp 10 platform comes with its own query languages, WarpScript, designed to help customers manipulate geo time series, providing more than 1000 off the shelf functions to find outliers, prepare data for machine learning applications, and several frameworks for the most commons operations (alignment, interpolation, sliding window operations, reduction between several pieces of information). The FLoWS language provide the same functions as WarpScript with a functional syntax.

A Warp 10 database can send data to another Warp 10 database easily (with or without preprocessing). For efficiency, a local Warp 10, acting as a concentrator for small devices, needs around 300MB of RAM on any hardware running Linux and a JVM.

The more powerful is your hardware, the least intermediate Warp 10 servers you need.

Small microcontrollers

Your customer, or you as a supplier, needs to install Warp 10 on-premise on an intermediate server, because your device won't keep a large history, and cannot support a loss of internet connection. This local Warp 10 can be as small as raspberry-like hardware. The amount of free space can be as little as 8 GB (to save locally several weeks of data at 200 datapoints per second). This intermediate server must be connected to an NTP server for time accuracy.

intermerdiate warp 10

Microcontrollers without an IP stack

You can use RS485, or other low-speed buses, with various protocol stacks on top, from the simplest Modbus rtu to BACnet mstp.

These devices can connect directly to Warp 10, but the machine hosting Warp 10 must have some specific plugins. Please contact SenX directly for support.

Microcontrollers with an IP stack

As long as your microcontroller can send a UDP datagram, or open a TCP socket, things are simpler, and you don't need any additional software.

You can set up Warp 10 UDP or TCP plugins to decode and push data into Warp 10, with a few second caches if needed for best performances.

Using TCP or UDP plugin, the payload format depend on you. WarpScript has functions to convert bytes to standard data types, or you can use some standardized payloads (Thrift, Protobuf, JSON).

Microcontrollers with an IP stack and a bit more power

As long as your device can connect using an SSL certificate, you can remove the local Warp 10 server, and push datapoints directly to your customer's Warp 10 instance, through HTTPS or MQTTs. If your customer has an MQTT broker, Warp 10 can subscribe with MQTT extension.

Our advice is to keep a local Warp 10 in the following cases:

  • your customer asks you for complex retention policies (support several months offline, do operations on the fly on data before pushing them)
  • you have strong bandwidth or communication cost constraints. In this case, the local Warp 10 can handle time series compression more efficiently than any other solution.

What will the local Warp 10 instance do?

Once data is stored in a local Warp 10 instance, you are free to implement what your customer wants. For example:

  • Stream data with 10 seconds chunks pushed to their Warp 10 server.
  • Optimize biggest chunks of data (COMPACT, LTTB) to save bandwidth and disk space.
  • Push subsampled data every 30 minutes, but push alarms in real-time, with all the data 10 minutes before the alarm at maximum time resolution.

We can help you to make your first WarpScript, then you'll be autonomous.

Depending on the customer requirements, the local Warp 10 configuration can be left open to your customer (give them the right to change anything in the /opt/warp10 directory). The bare minimum configurations your customer must be able to change are:

  • Warp 10 exec endpoint, ie: https://warp.customercompany.com/api/v0/exec
  • Warp 10 write token: ASCII string, ie "2HqQneKuMMnpI_nNkTWHlyLzGungAMEswOG.s6Qy4jI9ob_Y765noq7es92tNdfPVKeDoJGRZWMa2EwPi2bbjc80dH6wMU9BjuiABSw6aDoIuqCV0mf.sF".

Linux or docker capable high-end PLC, industrial PC

intermerdiate warp 10

These high-end hardwares remove the need for an intermediate local Warp 10 server. They can connect directly through internet to your customer's Warp 10 instance (see industrial dataloggers below). But if you need to support a long offline period or set up complex retention policies, or if you want to optimize communications, installing Warp 10 locally is the best solution.

From the software point of view, Warp 10 will be installed as a docker image, or natively. Your local software (codesys for example) must be able to open a local TCP port, or do an HTTP request, to push data to the local Warp 10.

Industrial dataloggers

Managing long data history and offline periods is already your core business. Your device already timestamps data precisely. Making an HTTPS request is not a problem either. You do not need local Warp 10 instance, and you can send data as Geo Time Series directly to your customer's Warp 10 instance.

What you need to do:

  • Provide a way for your customer to specify a Warp 10 endpoint, ie: https://warp.customer.com/api/v0/update
  • provide a way for your customer to specify a Warp 10 write token, which is a long ASCII string, ie:"2HqQneKuMMnpI_nNkTWHlyLzGungAMEswOG.s6Qy4jI9ob_Y765noq7es92tNdfPVKeDoJGRZWMa2EwPi2bbjc80dH6wMU9BjuiABSw6aDoIuqCV0mf.sF"
  • Provide a way to name channels, because it is easier to read "engine_temperature" than "blk3.a4-20.ch5" in the results.
  • Provide a way for your customer to configure data retention and sync policies: should local data be erased after successful upload? In case of internet loss, when the connection is up again, should most recent data be pushed before older data?
  • Provide a way for your customer to customize data with extra labels (global, or per channel). Labels are a key value structure, represented as a text string, such as {label1=valueoflabel1,label2=xyz,zone=331,boat=mmsi}
  • Serialize data to push them in an optimized way (channel per channel). See examples and recommendations below. Please read the page GTS input format, and contact us to make sure there are no modeling errors, we'll be happy to help you.
  • Do an HTTP POST request to the Warp 10 update endpoint. You may limit the request size up to your available memory, or use HTTP chunked requests (aka Transfer-Encoding: chunked header) to stream data. You also need to add the write token string as X-Warp10-Token: TOKEN_WRITE header. Your device should be able to do the equivalent of:
curl -H 'X-Warp10-Token: TOKEN_WRITE' -H 'Transfer-Encoding: chunked' -T METRICS_FILE 'https://HOST:PORT/api/v0/update'
  • Keep track of successful requests to avoid pushing twice the same thing.
  • If the connection is lost while transferring, re-push the full data chunk (Warp 10 will handle that).
  • If you can, compress the request body and add Content-Type: application/gzip header. As input format is text-based, compression is efficient.
  • There is no size limit to the HTTP request payload on the Warp 10 side.

Examples and recommendations to serialize data

Warp 10 is a time series database and not an SQL-based one. So, you do not need to align timestamps between channels or push values on a regular period.

We strongly advise you to push only value changes for each channel + periodic heartbeat values. GTS Input format to push to api/v0/update API is a simple text format, with one datapoint per line. The most efficient way is to group data per channel, to avoid switching channels every line.

#timestamp in us / latitude : longitude / elevation  classname{label1=x,label2=yy} value you want to store
1642770058482091/48.41922:4.4411/88 AccelerationX{deviceid=w34} -2.114
=1642770058582091/48.41922:4.4411/88 -2.115
=1642770058642090/48.41922:4.4411/88 -2.116
=1642770058882091/48.41922:4.4411/88 -2.115
1642770058482091// Temperature{deviceid=w34} 46.0
=1642780058582091// 45.5
=1642780058642090// 44.0
=1642780058882091// 43.0

Location is designed to store GPS location efficiently. But it just needs to be stored alongside one channel, it is a waste of resources to store it on several channels, except if your customer specifically requested it.

It is also a waste of resources to repeat the location information every 10ms if it updates only every second.

As you will push only value changes, the customer may doubt that the system is alive if all the channels remain constants. So, we strongly advise you to store a few virtual channels :

  • heartbeat: just push a boolean every second (name and frequency to be discussed with your customer).
1642804230556621// heartbeat{deviceid=w34} true
=1642804231556621//  true
=1642804232556621//  true
=1642804233556621//  true
=1642804234556621//  true
  • datalogger internals: It could be remaining disk space, CPU usage, internal battery state, power loss events... To be discussed with your customer.



As Warp 10 was designed for the IoT we have written many articles about it on our blog.

Explore the list thanks to IoT and Industry tags.



Contact our team to discuss how to integrate Warp 10 into your industrial IoT project.