Fetch
The Fetch API allows to recover raw GTS data in a extremely quick and efficient way.
The HTTP endpoint used by the Fetch API is https://HOST:PORT/api/vX/fetch
, where HOST:PORT
is a
valid endpoint for the Warp 10 instance and vX
is the version of the API you want to use
(currently v0
).
The Fetch API must be accessed using the GET method.
Note that the amount of data that can be retrieved via the /fetch
endpoint is not limited in any way. You can use this endpoint to dump all the data accessible with a given token. The time needed for data retrieval is dependent on the size of the retrieved dataset.
Request Header
To be authenticated you need to add a X-Warp10-Token
header with a valid READ token.
Header | Description |
---|---|
X-Warp10-Token |
A valid READ token |
As modifying request headers can be difficult with certain HTTP clients, there is a fallback authentication mode: if
headers can't be used, the token can be passed in a request parameter token
.
Request parameters
The Fetch API request understands the following parameters:
-
selector
: the selector is a string that allows to select one or several GTS. Its composed of the concatenation of a classname selector and a labels selector.The class name selector can be a sequence of characters representing an exact match if it starts with an
=
, or a regular expression that the class name must match if it starts with~
. When there is no ambiguity, the leading=
can be omitted.The labels selector is delimited by curly braces (
{}
) and is comma separated list of individual label selectors. Each of these label selectors is composed of the label name and the value of the associated selector. Those selectors can also be exact matches if they start with=
or a regular expression if they start with~
.Example:
~warp.*{freq~2.*,steps=100}
It selects all GTS whose classname begin with warp and whose label steps has a value of 100 and whose label freq begins with 2.
The class and label elements of the selector can be percent encoded.
-
format
: the format of the response data. Current available formats are:text
: ready for ingestion in another Warp 10 instance.fulltext
: ready for ingestion in another Warp 10 instance, but less optimizedjson
: ready for web requests. See Json Output Formattsv
: tabulation separated values, with only timestamp and valuesfulltsv
: tabulation separated values, with classname and labels on each linepack
: GTS with wrapped GTS in the value: compressed object represented as base64 string, ready to UNWRAP.
-
dedup
: if this parameter is true, sequences of successive datapoints with the same locations and value are compressed in the response, giving only the first and the last datapoints in the sequence. It is the same behavior as RANGECOMPACT WarpScript function.Example:
-
Response to request without
dedup
:1434590504758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42 1434590288758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42 1434590072758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42 1434589856758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42 1434589640758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 43 1434589424758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 44
-
Response to request with
dedup
1434590504758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42 1434589856758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42 1434589640758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 43 1434589424758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 44
-
-
Fetching interval parameters: either
start
andstop
: the start and end timestamps defining the interval of the GTS to fetch. They are both in ISO8601 format, i.e. YYYY-MM-DDTHH:MM:SS.SSSSSSZ. Only those readings whose timestamps lie between those two timestamps (most recent inclusive, oldest exclusive) will be retained.
Or
-
now
andtimespan
: instead of the basic interval search, with the start and end timestamps in ISO8601 format, you can use two alternative formats for the interval parameters:-
If you want to fetch readings whose timestamps lie in a specific timespan before a timestamp (for example all the readings in the last minute) you can use the end timestamp (in time units since the Unix epoch) as
now
parameter and the timespan (in platform time units) astimespan
. -
If you want to recover the last
n
readings before a given instant, you can use the instant timestamp (in microseconds since the Unix epoch) asnow
parameter and-n
astimespan
.
You can use
now
as the value of thenow
parameter to dynamically retrieve the current time.The
timespan
parameter can be specified as an ISO-8601 duration. Note that supported durations cannot contain years or months. -
-
showattr
can be set totrue
to display the attributes of the retrieved Geo Time Series. -
sortmeta
can be set totrue
to sort the labels so they are always displayed in the same way for a given GTS. -
showerrors
can be set totrue
to display a special string starting with# ERROR:
at the end of the output if an error was encountered while fetching data. We cannot rely on the HTTP error mechanism since we may have already started outputting content when the error is encountered.
Multi Value output
Since Warp 10 v2.1, Warp 10 can handle multivalue in the value field.
Multi value is saved as binary, so the output is binary, encoded in base64 for text
and fulltext
format. For json
format, value is a string with encoded unicode characters.
- Input:
1// mvtest{} [ 4 4.5 -2.2]
- Output:
1// mvtest{} b64:eAgsCDABWjABKwA
This is very handy to ingest in another Warp 10 database, but if you plan to build a web application that must retrieve multivalues, use the exec endpoint to rebuild with WarpScript MVxx functions simple GTS from multivalues one. The output will be a ready to use json.
Example
GET /api/v0/fetch?now=1435091737000000×pan=-10&selector=io.warp10.test.sinusoide{steps=100}&format=fulltext HTTP/1.1
Host: host
Response
The format of the response depends on the format
parameter of the request.
-
fulltext
The fulltext format uses the GTS Input Format as output format.
Example:
1434590504758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.6133061918698982 1434590288758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.9228427144511169 1434590072758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.1301889411087915 1434589856758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.9909979074466949 1434589640758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.38860822449533416 1434589424758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.7875576742396682 1434589208758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.8009024241854527 1434588992758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.36827736651082893 1434588776758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.993699252984253 1434588560758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.15193398008971615
-
text
The text format uses a less verbose version of the GTS Input Format as output format. In this format the data for each GTS is grouped and classname and labels are only written in the first line of each group.
Example:
1434590504758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.6133061918698982 =1434590288758000// 0.9228427144511169 =1434590072758000// 0.1301889411087915 =1434589856758000// -0.9909979074466949 =1434589640758000// 0.38860822449533416 1434590504758000/48.44348858110607:-4.407284064218402/ io.warp10.drill.test.sinusoide{location=true,steps=100,.app=io.warp10.drill.test} -0.6133061918698982 =1434590288758000/48.44348858110607:-4.407284064218402/ 0.9228427144511169 =1434590072758000/48.44348858110607:-4.407284064218402/ 0.1301889411087915 =1434589856758000/48.44348858110607:-4.407284064218402/ -0.9909979074466949 =1434589640758000/48.44348858110607:-4.407284064218402/ 0.38860822449533416
Tooling
curl
command has no limits... However, you can use WarpStudio tool to retrieve data from your platform, or to help you to build the curl command:
Next steps
Read more about fetching data on the blog