Loading

What is a GTS?

All over the Warp 10 resources (here and on the blog), you will read about GTS.

GTS stands for Geo Time Series, and it is the most important subject you need to understand to use Warp 10 efficiently.

Roughly speaking, you can represent a GTS as a container with 5 columns:

  • timestamp
  • latitude
  • longitude
  • elevation
  • the value you want to store

GTS are indexed by time, and by their metadata:

  • the classname
  • the labels

Space indexation is also possible, although not activated by default.

gts

Once created in the database, you cannot add labels or change classname to an existing GTS. If you change a label, you reference a new GTS. That's why attributes exist: they are a kind of post-it note you can stick on each GTS to remember something, and they can be easily changed.

The classic input format for Warp 10 is a text payload pushed to an HTTPS endpoint, with one value per line:

<timestamp_in_platform_timeunit>/<latitude>:<longitude>/<elevation>  <classname>{label1=x,label2=yy} <value_you_want_to_store>
1642770058482091/48.41922:4.4411/88 AccelerationX{deviceid=w34} -2.114
1642771609112783/48.41914:4.4410/88 Temperatures{system=hh4,deviceid=w34} [ 48.4 51.0 47.9 true false true ]
1642772543842961/48.41914:4.4411/88 Alert.maxPitchAngle{deviceid=w34} true
1642772550799612/48.41914:4.4411/88 WStatus{deviceid=w34} 4
1642772550799612/48.41914:4.4411/88 OperatorMessage{deviceid=w34} "bad reaction with k23 mix, manual purge"

There are lots of alternatives to HTTP built into Warp 10 to ingest data (from raw UDP datagrams to MQTT or a Kafka consumer).

There are lots of possible value types. You can even store spatio-temporal tensors, there is no dimension limit. SenX will be happy to help you to solve complex data modeling issues.

There is no schema

geo

You do not need to configure Warp 10 with all the GTS classname or labels at startup. Once Warp 10 is running, just push your GTS to the /api/v0/update endpoint, Warp 10 directory will index your classname and labels.

Location is NOT required

geo

You can use the Warp 10 GTS model without any location information. You just don't write latitude, longitude, or elevation in the Warp 10 input.

You can write only latitude and longitude... Or just elevation.

In this case, your input will look like:

1642770058482091// AccelerationX{deviceid=w34} -2.114
1642770135251786// AccelerationX{deviceid=w34} -2.0
1642770143080815/48.41914:4.4411/ AccelerationX{deviceid=w34} -1.4
1642770150165608// Alert.Temperature{deviceid=w34}  true
1642770150165608//88 AccelerationX{deviceid=w34} -1.22
1642770175336130// AccelerationX{deviceid=w34} -1.23

Depending on your use case, you may not need these pieces of information, or you may need to write them once in one GTS, but not in other GTS from the same device. Or you store data at 100 Hz while your GPS device output one point every second, so you store geo information on one point out of 100.

Ultimately, every missing Geo information saves server resources, so Warp 10 lets you free to optimize your inputs.

Geo is for Geo only

geo

Warp 10 uses a highly optimized format to store latitude and longitude. It means you cannot store in these fields something else than values included in [-90;90] range for latitude, and [-180;180] range for longitude. Position precision is about 1 cm at the equator, worst case. Precision is better everywhere else on the earth.

Elevation information might be used to store any 64 bits integer number, so you can use the unit you want (default unit is millimeter).

Regular timestamps are NOT required

ts

GTS are independant from one another, so you don't need to align timestamps between different GTS or make regular timestamps. There is no point to repeat the same value if it does not change in time. So, in Warp 10 inputs, you can have GTS that change every 10ms (noisy sensor) mixed with GTS that change occasionally (physical events, or computed alerts).

In this case, your input would look like:

1642770058482091// AccelerationX{deviceid=w34} -2.114
1642770135251786// AccelerationX{deviceid=w34} -2.0
1642770143080815// AccelerationX{deviceid=w34} -1.4
1642770150165608// Alert.Temperature{deviceid=w34}  true
1642770150165608// AccelerationX{deviceid=w34} -1.22
1642770175336130// AccelerationX{deviceid=w34} -1.23

Note that Warp 10 provides boundary fetches in the FETCH function, so you can request data in a time range + the nearest points (even if the nearest points are far away in the past or future). Alerts, status, hand switches, are typically information that does not change often.

The only use case where you need to repeat information is to track communication loss. The timestamp is the information. In this case, you can create a GTS named "HeartBeat", pushing true (boolean value) every second, and process it later with WarpScript to compute data loss.

The disk storage behavior is "overwrite"

ts

On persistent backends, storing duplicate timestamps is not possible by design. If you push two times the same timestamp, the last value will overwrite the previous ones.

1642770058000000// windSpeed{deviceid=w34} 37.5
1642770058000000// windSpeed{deviceid=w34} 38.0
1642770058000000// windSpeed{deviceid=w34} 37.0

Stored value for windSpeed of device w34 will be 37.0, because the timestamps are identical.

Our advice is to take advantage of the microsecond timestamps. If your system cannot provide anything else than a one-second resolution timestamp, you may artificially add a microsecond between each measurement, if you want to keep them all.

The overwrite behavior can be somehow prevented if you use tokens with time limits: You can generate write tokens that cannot write in the future or x minutes before the current time. It can save your data integrity if you use embedded systems without any reliable RTC, or with timers silicon bugs.

Note that the in-memory backend allows duplicate timestamps. Warp 10 can use in-memory backends if you need to reach 100 million datapoints/second on a single server, or if you need an accelerator for the last hours of data together with spinning disks storage.

Data are typed

ts

The first datapoint defines the GTS type. GTS type cannot be changed afterward in the storage. Be careful with languages that round "45.0" to "45"...

1642775428399823// petrol{unit=ton,deviceid=w34} 37
1642775435230182// petrol{unit=ton,deviceid=w34} 36.97
1642775442118000// petrol{unit=ton,deviceid=w34} 36.96

In the example above, petrol will be rounded to the nearest integer, because the first datapoint of your GTS was an integer.

Multivalue data are not typed, the following input will be stored as it is:

1642775428399823// gyroscope{deviceid=w34} [ -1.0 4.44 1 true true ]
1642775435230182// gyroscope{deviceid=w34} [ true false ]
1642775442118000// gyroscope{deviceid=w34} [ "systemError, reboot" ]

Where can GTS be used?

Warp 10 comes with lots of tools to convert manipulate GTS from Spark, Python, and other ecosystems.

ecosystem

To go on with GTS, once you are done with the first tutorials, you can follow the "Master GTS" tutorial.

We also talk about GTS on the blog.