Fetch

The Fetch API allows to recover raw GTS data in a extremely quick and efficient way.

The HTTP endpoint used by the Fetch API is https://HOST:PORT/api/vX/fetch, where HOST:PORT is a valid endpoint for the Warp 10 instance and vX is the version of the API you want to use (currently v0).

The Fetch API must be accessed using the GET method.

Note that the amount of data that can be retrieved via the /fetch endpoint is not limited in any way. You can use this endpoint to dump all the data accessible with a given token. The time needed for data retrieval is dependent on the size of the retrieved dataset.

Request Header

To be authenticated you need to add a X-Warp10-Token header with a valid READ token.

As modifying request headers can be difficult with certain HTTP clients, there is a fallback authentication mode: if headers can't be used, the token can be passed in a request parameter token.

Request parameters

The Fetch API request understands the following parameters:

• selector: the selector is a string that allows to select one or several GTS. Its composed of the concatenation of a classname selector and a labels selector.

  The class name selector can be a sequence of characters representing an exact match if it starts with an =, or a regular expression that the class name must match if it starts with ~. When there is no ambiguity, the leading = can be omitted.

  The labels selector is delimited by curly braces ({}) and is comma separated list of individual label selectors. Each of these label selectors is composed of the label name and the value of the associated selector. Those selectors can also be exact matches if they start with = or a regular expression if they start with ~.

  Example:

  ~warp.*{freq~2.*,steps=100}

  It selects all GTS whose classname begin with warp and whose label steps has a value of 100 and whose label freq begins with 2.

  The class and label elements of the selector can be percent encoded.

• format: the format of the response data. Current available formats are:

  • text: ready for ingestion in another Warp 10 instance.
  • fulltext: ready for ingestion in another Warp 10 instance, but less optimized
  • json: ready for web requests. See Json Output Format
  • tsv: tabulation separated values, with only timestamp and values
  • fulltsv: tabulation separated values, with classname and labels on each line
  • pack: GTS with wrapped GTS in the value: compressed object represented as base64 string, ready to UNWRAP.

• dedup: if this parameter is true, sequences of successive datapoints with the same locations and value are compressed in the response, giving only the first and the last datapoints in the sequence. It is the same behavior as RANGECOMPACT WarpScript function.

  Example:

  • Response to request without dedup:

    1434590504758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42
    1434590288758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42
    1434590072758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42
    1434589856758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42
    1434589640758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 43
    1434589424758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 44
    

  • Response to request with dedup

    1434590504758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42
    1434589856758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 42
    1434589640758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 43
    1434589424758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 44
    

• Fetching interval parameters: either

  • start and stop: the start and end timestamps defining the interval of the GTS to fetch. They are both in ISO8601 format, i.e. YYYY-MM-DDTHH:MM:SS.SSSSSSZ. Only those readings whose timestamps lie between those two timestamps (most recent inclusive, oldest exclusive) will be retained.

  Or

  • now and timespan: instead of the basic interval search, with the start and end timestamps in ISO8601 format, you can use two alternative formats for the interval parameters:

    1. If you want to fetch readings whose timestamps lie in a specific timespan before a timestamp (for example all the readings in the last minute) you can use the end timestamp (in time units since the Unix epoch) as now parameter and the timespan (in platform time units) as timespan.

    2. If you want to recover the last n readings before a given instant, you can use the instant timestamp (in microseconds since the Unix epoch) as now parameter and -n as timespan.

    You can use now as the value of the now parameter to dynamically retrieve the current time.

    The timespan parameter can be specified as an ISO-8601 duration. Note that supported durations cannot contain years or months.

• showattr can be set to true to display the attributes of the retrieved Geo Time Series.

• sortmeta can be set to true to sort the labels so they are always displayed in the same way for a given GTS.

• showerrors can be set to true to display a special string starting with # ERROR: at the end of the output if an error was encountered while fetching data. We cannot rely on the HTTP error mechanism since we may have already started outputting content when the error is encountered.

Multi Value output

Since Warp 10 v2.1, Warp 10 can handle multivalue in the value field.

Multi value is saved as binary, so the output is binary, encoded in base64 for text and fulltext format. For json format, value is a string with encoded unicode characters.

• Input: 1// mvtest{} [ 4 4.5 -2.2]
• Output: 1// mvtest{} b64:eAgsCDABWjABKwA

This is very handy to ingest in another Warp 10 database, but if you plan to build a web application that must retrieve multivalues, use the exec endpoint to rebuild with WarpScript MVxx functions simple GTS from multivalues one. The output will be a ready to use json.

Example

GET /api/v0/fetch?now=1435091737000000&timespan=-10&selector=io.warp10.test.sinusoide{steps=100}&format=fulltext HTTP/1.1
Host: host

Response

The format of the response depends on the format parameter of the request.

• fulltext

  The fulltext format uses the GTS Input Format as output format.

  Example:

  1434590504758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.6133061918698982
  1434590288758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.9228427144511169
  1434590072758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.1301889411087915
  1434589856758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.9909979074466949
  1434589640758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.38860822449533416
  1434589424758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.7875576742396682
  1434589208758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.8009024241854527
  1434588992758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.36827736651082893
  1434588776758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} 0.993699252984253
  1434588560758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.15193398008971615
  

• text

  The text format uses a less verbose version of the GTS Input Format as output format. In this format the data for each GTS is grouped and classname and labels are only written in the first line of each group.

  Example:

    1434590504758000// io.warp10.test.sinusoide{steps=100,.app=io.warp10.drill.test} -0.6133061918698982
    =1434590288758000// 0.9228427144511169
    =1434590072758000// 0.1301889411087915
    =1434589856758000// -0.9909979074466949
    =1434589640758000// 0.38860822449533416
    1434590504758000/48.44348858110607:-4.407284064218402/ io.warp10.drill.test.sinusoide{location=true,steps=100,.app=io.warp10.drill.test} -0.6133061918698982
    =1434590288758000/48.44348858110607:-4.407284064218402/ 0.9228427144511169
    =1434590072758000/48.44348858110607:-4.407284064218402/ 0.1301889411087915
    =1434589856758000/48.44348858110607:-4.407284064218402/ -0.9909979074466949
    =1434589640758000/48.44348858110607:-4.407284064218402/ 0.38860822449533416
  

• json See Json Output Format

Tooling

curl command has no limits... However, you can use WarpStudio tool to retrieve data from your platform, or to help you to build the curl command:

Next steps

Read more about fetching data on the blog

• FETCHez la data !
• Fetch the first data point in Warp 10
• Fetch data within a geographical area
• Fetch data within a temporal or a count range