Deleting data

The Delete API is used to erase data from the Warp 10 Storage Engine. The Geo Time Series to delete are chosen via a selector, and the time interval to erase is defined by optional parameters.

The HTTP endpoint used by the Delete API is https://HOST:PORT/api/vX/delete, where HOST:PORT is a valid endpoint for the Warp 10 cell and vX is the version of the API you want to use (currently v0).

The Delete API must be accessed using the GET method.

The deletion feature is a very important one, especially with regulations such as the GDPR in Europe. The Distributed Warp 10 Storage Engine has the ability to delete millions of datapoints per second.

Request Header

To be authenticated you need to add a X-Warp10-Token header with a WRITE token valid for deletions, i.e. a WRITE token with the same value for owner and producer.

Request parameters

The /delete endpoint accepts the following parameters:

• selector: the selector to use for identifying Geo Time Series to delete, same syntax as for /fetch and /find.

• delete interval parameters:

  • start and end: the start and end timestamps defining the interval of the GTS to delete. They are both in ISO8601 format, i.e. YYYY-MM-DDTHH:MM:SS.SSSSSSZ or in microseconds since the Unix epoch. Only those readings whose timestamps lie between those two timestamps (most recent inclusive, oldest exclusive) will be deleted. Warning: For security purpose, both parameters (start and end) should be defined, with start > end.

• deleteall: should be set to allow deleting all data (start and end not defined). If this parameter is set, the Geo Time Series will also be removed from the Directory.

• minage: in the distributed version Warp 10, this parameter can be set to a number of milliseconds which will be checked for every datapoint to remove, if a datapoint was written less than this number of milliseconds ago in HBase then it won't be deleted.

• dryrun: when set, the /delete endpoint will simply identify the Geo Time Series which will be selected for the delete operation without actually erasing any data.

• showerrors can be set to true to display a special string starting with # ERROR: at the end of the output if an error was encountered while fetching data. We cannot rely on the HTTP error mechanism since we may have already started outputting content when the error is encountered.

Response

The call to /delete will output the list of Geo Time Series affected by the delete operation. In the case of the Distributed version of Warp 10, the output of a Geo Time Series means that the deletion request was pushed into Kafka. For the Standalone version of Warp 10, a GTS will be listed once the deletion has been performed.

Delete examples

Consider the following dataset

1440000000000000// toto{a=42,b=42} 42
1440000000000000// titi{a=42,b=42} 42
1441000000000000// toto{a=4,b=4} 4
1441000000000000// titi{a=4,b=4} 4
1442000000000000// toto{a=42,b=42} 42
1442000000000000// titi{a=42,b=42} 42
1443000000000000// toto{a=4,b=4} 4
1443000000000000// titi{a=4,b=4} 4
1444000000000000// toto{a=42,b=42} 42
1444000000000000// titi{a=42,b=42} 42

it contains 4 Geo Time Series:

• toto{a=42,b=42}
• titi{a=42,b=42}
• toto{a=4,b=4}
• titi{a=4,b=4}

The timestamps used have the following human representations:

• Initial timestamp 1440000000000000 is 2015-08-19T16:00:00 in ISO8601
• Final timestamp 1444000000000000 is 2015-10-04T23:06:40 in ISO8601

The following deletions can be performed:

• Delete all the points from the toto{a=42,b=42} GTS:

GET /api/v0/delete?deleteall&selector=toto{a=42,b=42} HTTP/1.1

Resulting dataset via fetch API

1440000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42
1441000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1441000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4
1442000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42
1443000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1443000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4
1444000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42

• Delete all the points from the GTS having a label a with a value 42:

GET /api/v0/delete?deleteall&selector=~.*{a=42} HTTP/1.1

Resulting dataset via fetch API

1441000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1441000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4
1443000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1443000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4

• Delete all the points in September 2015 from the GTS having a classname toto:

GET /api/v0/delete?selector=toto{}&start=2015-09-01T00:00:00Z&end=2015-09-30T23:59:59.999Z HTTP/1.1

Resulting dataset via fetch API

1440000000000000// toto{a=42,b=42,.app=ovh.hgg.test.01} 42
1440000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42
1441000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1441000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4
1444000000000000// toto{a=42,b=42,.app=ovh.hgg.test.01} 42
1444000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42

• Delete all points after September 2nd at noon from all GTS with a label b with value 4, using regex for the classname:

GET /api/v0/delete?selector=~t.t.{b=4}&start=2015-09-02T12:00:00Z HTTP/1.1

Resulting dataset via fetch API

1440000000000000// toto{a=42,b=42} 42
1440000000000000// titi{a=42,b=42} 42
1441000000000000// toto{a=4,b=4} 4
1441000000000000// titi{a=4,b=4} 4
1442000000000000// toto{a=42,b=42} 42
1442000000000000// titi{a=42,b=42} 42
1444000000000000// toto{a=42,b=42} 42
1444000000000000// titi{a=42,b=42} 42