GTS Input Format
- Ingesting data in Warp 10 can be done via the Ingress endpoint. The body of the request must respect the GTS input format described below.
- Ingestion can also be done through a websocket. The messages must respect the GTS input format described below.
- The WarpScript PARSE function also expects a multiline string that follow the GTS input format described below.
- By default, fetching data from Warp 10 will also produce an output that follow the GTS input format described below.
The GTS input format evolves with Warp 10, but there is no breaking change since several years.
Lines
The GTS input format has one line per timestamp and per data pushed into the platform.
Each line has the following format:
TS/LAT:LON/ELEV NAME{LABELS}{ATTRIBUTES} VALUE
where each element has the following syntax and meaning:
| Parameter | Description |
|---|---|
TS | Timestamp of the reading, in the configured time units since the Unix Epoch. Default configuration is microsecond unit. If omitted, the time of the data ingestion will be used as the time of each reading |
LAT:LON | Optional geographic coordinates of the reading, using WGS84 |
ELEV | Optional elevation of the reading, in millimeters |
NAME | Class name of the reading as a URL encoded UTF-8 character string. The encoding of character { (Unicode LEFT CURLY BRACKET, 0x007B) is MANDATORY. |
LABELS | Comma separated list of labels, using the syntax key=value where both key and value are URL encoded UTF-8 character strings. If a key or value contains , (Unicode COMMA, 0x002C), } (Unicode RIGHT CURLY BRACKET, 0x007D) or = (Unicode EQUALS SIGN, 0x003D), those characters MUST be encoded. |
ATTRIBUTES | Optional comma separated list of attributes, using the syntax key=value where both key and value are URL encoded UTF-8 character strings. If a key or value contains , (Unicode COMMA, 0x002C), } (Unicode RIGHT CURLY BRACKET, 0x007D) or = (Unicode EQUALS SIGN, 0x003D), those characters MUST be encoded. Attributes is a mutable value, setting them with update endoint is possible but not recommended. Prefer meta endpoint. You have to set ingress.parse.attributes = true in the config file |
VALUE | The value of the reading. It can be of one of four types: LONG, DOUBLE, BOOLEAN, STRING. Since Warp 10 v2.1, Value can also be BINARY or MULTIVALUE |
Optionnal parameters can be removed from input format. The simplest line is:
TS// NAME{} VALUE
Note that empty lines and comment lines (lines starting with #) are ignored. Trailing and leading spaces must be removed.
Values
LONG
A LONG value is represented as a sequence of digits optionally prefixed with a '+' or '-' sign. The values that can be represented span from -2^63 to 2^63-1.
1234567890
DOUBLE
A DOUBLE value is represented as a sequence of digits, a mandatory decimal separator ('.'), and a mandatory sequence of digits following the separator. Scientific notation can also be used. The values that can be represented are those representable in the 64 bits IEEE754 double precision floating point format.
3.14159265359
-45.5e6
BOOLEAN
A BOOLEAN value is represented as either T (for true) or F (for false). Note that 't', 'f', 'true', 'false' are also accepted.
T
F
STRING
A STRING value is represented as a URL encoded UTF-8 character string enclosed in single quotes or double quotes.
'caf%C3%A9'
HH Code
HH Code is the LONG representation of latitude and longitude. This value format starts with HH: then latitude, then :, then the longitude.
HH:48.442172:-4.414402
QUATERNIONS
Quaternions are a number system that extends the complex numbers. Each quaternion dimension is encoded with 16 bits, the final representation is also a LONG.
Q:w:x:y:z
BINARY
Since Warp 10 v2.1, you can store binary. Maximum size is linked to the max.encoder.size, which is also linked to kafka setup on distributed architecture. The binary data can be represented as base64 or hexadecimal string.
b64:aGVsbG8gV2FycCAxMAo=
hex:68656c6c6f2057617270203130
MULTI VALUE
Value is a nested set of lists, each one being a space separated list, enclosed in [ ].
[1 2 3.5]
When your datas are well known and time aligned, multivalue save disk and performances. The schema (data name for indices) could be saved as an attribute of the GTS.
The multivalue is compressed and saved as binary. Ingestion and fetch of multivalue is far more performant than ingestion of several GTS, but if you have enough disk space, or working with an in memory instance, you can speed up again by explicitly remove the compression. Just add an exclamation mark after the opening bracket to skip the compression step.
[!1 2 3.5]
Value of multivalues can also be an unnamed GTS, using the following syntax:
[ value ]
[ timestamp/value ]
[ timestamp/latitude:longitude/value ]
[ timestamp//elevation/value ]
[ timestamp/latitude:longitude/elevation/value ]
Since Warp 10 v2.1, complex data modelisation is possible within a GTS value: ingestion of the following lines in Warp 10 is possible, but you need to remember the schema to decode it later:
#value: speed, magnetic heading in degree, true heading, pilot destination
# and latest computed arrival time (value 15 is the error margin in percent)
1566893340372549/48.8:-4.15/124 shipNavigationData{mmsi=311000632} [8.1 152 153 1568189777837909/40.6:-74/15 ]
1566893344654882/48.81:-4.147/124 shipNavigationData{mmsi=311000632} [8.2 151 152 1568189745655509/40.6:-74/14 ]
Be careful, strings in multi value must be URL-encoded. For performance reasons, / and inside string will break the parsing.
There are lots of multivalue helper functions, starting with MV (MultiValue). These functions allow an easy reconstruction of several simple GTS from a multivalue one.
Special timestamps
No timestamp means "take the current Warp 10 machine timestamp right now while parsing". It has a performance cost, but it is really usefull for slow devices without RTC. It should not be used on several consecutive lines:
#Do NOT do this ! Timestamp delta will depend on your hardware ! Prefer relative timestamps...
// test{} T
=// T
=// F
=// T
=// T
=// T
=// T
=// F
=// F
=// T
=// T
There are cases in IoT where an absolute timestamp is not available but devices can only create relative timestamps from their startup date.
Such cases can be handled by Warp 10 by using a special syntax for the TS element of the input format to specify relative timestamps. This syntax is T+xxx or T-xxx where xxx is the delta from the base now timestamp in time units.
The now timestamp is assumed to be the time of the request unless the X-Warp10-Now request header was used. This header can specify a value for now, either as an absolute timestamp expressed in time units or as a time units delta (prefixed by either + or -) from the current timestamp. The special value * can also be used to retrieve the current timestamp each time a timestamp is needed.
Example: a temperature sensor read temperature every 20 seconds, and transmits every 2 minutes:
#that is a good practice for low-energy devices:
# keep a local history to minimize transmissions, rely on database absolute timestamp.
T-120000000// temperature{sensor=xx} 42.12
=T-100000000// 42.09
=T-80000000// 41.99
=T-60000000// 41.79
=T-40000000// 41.54
=T-20000000// 41.3
=// 41.21
Relative timestamps are correctly handled by the datalog replication mechanism.
Continuation lines
When pushing multiple datapoints for the same Geo Time Series, the first line can have the syntax described above (TS/LAT:LON/ELEV CLASS{LABELS} VALUE), but the following can use the more compact syntax =TS/LAT:LON/ELEV VALUE which will also speed up ingestion as the class and labels will not be parsed for each value.
Request Headers
The Warp 10 ingress token MUST be passed as the value of the X-Warp10-Token HTTP header.
If the request body is compressed, the Content-Type header MUST be set to the value application/gzip.
Example 1
POST /api/v0/update HTTP/1.1
Host: host
X-Warp10-Token: TOKEN
Content-Type: text/plain
1380475081000000// foo{label0=val0,label1=val1} 123
/48.0:-4.5/ bar{label0=val0} 3.14
1380475081123456/45.0:-0.01/10000000 foobar{label1=val1} T
Example 2
Performance tip: Gather your data together. If you do one ingress call per line, the http overhead will be huge. Ingress size has no limit. You can curl a 200 GB file, the ingestion rate will be very high. Gathering your data per channel will be even faster, because you don't need to repeat the class name and labels on each line. Just start the line with '='.
#no latitude, no longitude, no elevation, same channel for the following lines.
#starting a line with "=" means "keep previous names and labels from previous line".
1521444669000000// tool_speed{plant=paris,machine=47} 104.392492
=1521444669000020// 105.240093
=1521444669000040// 105.336131
=1521444669000060// 106.168065
Test easily your input format
Copy-paste your input in a multiline string, then use WarpScript PARSE function: