Deleting data
The Delete API is used to erase data from the Warp 10 Storage Engine. The Geo Time Series to delete are chosen via a selector, and the time interval to erase is defined by optional parameters.
The HTTP endpoint used by the Delete API is https://HOST:PORT/api/vX/delete
, where HOST:PORT
is a
valid endpoint for the Warp 10 cell and vX
is the version of the API you want to use (currently v0
).
The Delete API must be accessed using the GET method.
The deletion feature is a very important one, especially with regulations such as the GDPR in Europe. The Distributed Warp 10 Storage Engine has the ability to delete millions of datapoints per second.
Request Header
To be authenticated you need to add a X-Warp10-Token
header with a WRITE token valid for deletions, i.e. a WRITE token with the same value for owner
and producer
.
Header | Description |
---|---|
X-Warp10-Token |
A valid WRITE token |
Request parameters
The /delete
endpoint accepts the following parameters:
-
selector
: the selector to use for identifying Geo Time Series to delete, same syntax as for/fetch
and/find
. -
delete interval parameters:
start
andend
: the start and end timestamps defining the interval of the GTS to delete. They are both in ISO8601 format, i.e. YYYY-MM-DDTHH:MM:SS.SSSSSSZ or in microseconds since the Unix epoch. Only those readings whose timestamps lie between those two timestamps (most recent inclusive, oldest exclusive) will be deleted. Warning: For security purpose, both parameters (start
andend
) should be defined, withstart
>end
.
-
deleteall
: should be set to allow deleting all data (start
andend
not defined). If this parameter is set, the Geo Time Series will also be removed from the Directory. -
minage
: in the distributed version Warp 10, this parameter can be set to a number of milliseconds which will be checked for every datapoint to remove, if a datapoint was written less than this number of milliseconds ago in HBase then it won't be deleted. -
dryrun
: when set, the/delete
endpoint will simply identify the Geo Time Series which will be selected for the delete operation without actually erasing any data. -
showerrors
can be set totrue
to display a special string starting with# ERROR:
at the end of the output if an error was encountered while fetching data. We cannot rely on the HTTP error mechanism since we may have already started outputting content when the error is encountered.
Response
The call to /delete
will output the list of Geo Time Series affected by the delete operation. In the case of the Distributed version of Warp 10, the output of a Geo Time Series means that the deletion request was pushed into Kafka. For the Standalone version of Warp 10, a GTS will be listed once the deletion has been performed.
Delete examples
Consider the following dataset
1440000000000000// toto{a=42,b=42} 42
1440000000000000// titi{a=42,b=42} 42
1441000000000000// toto{a=4,b=4} 4
1441000000000000// titi{a=4,b=4} 4
1442000000000000// toto{a=42,b=42} 42
1442000000000000// titi{a=42,b=42} 42
1443000000000000// toto{a=4,b=4} 4
1443000000000000// titi{a=4,b=4} 4
1444000000000000// toto{a=42,b=42} 42
1444000000000000// titi{a=42,b=42} 42
it contains 4 Geo Time Series:
- toto{a=42,b=42}
- titi{a=42,b=42}
- toto{a=4,b=4}
- titi{a=4,b=4}
The timestamps used have the following human representations:
- Initial timestamp
1440000000000000
is2015-08-19T16:00:00
in ISO8601 - Final timestamp
1444000000000000
is2015-10-04T23:06:40
in ISO8601
The following deletions can be performed:
- Delete all the points from the
toto{a=42,b=42}
GTS:
GET /api/v0/delete?deleteall&selector=toto{a=42,b=42} HTTP/1.1
Resulting dataset via fetch API
1440000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42
1441000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1441000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4
1442000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42
1443000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1443000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4
1444000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42
- Delete all the points from the GTS having a label
a
with a value 42:
GET /api/v0/delete?deleteall&selector=~.*{a=42} HTTP/1.1
Resulting dataset via fetch API
1441000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1441000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4
1443000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1443000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4
- Delete all the points in September 2015 from the GTS having a classname
toto
:
GET /api/v0/delete?selector=toto{}&start=2015-09-01T00:00:00Z&end=2015-09-30T23:59:59.999Z HTTP/1.1
Resulting dataset via fetch API
1440000000000000// toto{a=42,b=42,.app=ovh.hgg.test.01} 42
1440000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42
1441000000000000// toto{a=4,b=4,.app=ovh.hgg.test.01} 4
1441000000000000// titi{a=4,b=4,.app=ovh.hgg.test.01} 4
1444000000000000// toto{a=42,b=42,.app=ovh.hgg.test.01} 42
1444000000000000// titi{a=42,b=42,.app=ovh.hgg.test.01} 42
- Delete all points after September 2nd at noon from all GTS with a label
b
with value 4, using regex for the classname:
GET /api/v0/delete?selector=~t.t.{b=4}&start=2015-09-02T12:00:00Z HTTP/1.1
Resulting dataset via fetch API
1440000000000000// toto{a=42,b=42} 42
1440000000000000// titi{a=42,b=42} 42
1441000000000000// toto{a=4,b=4} 4
1441000000000000// titi{a=4,b=4} 4
1442000000000000// toto{a=42,b=42} 42
1442000000000000// titi{a=42,b=42} 42
1444000000000000// toto{a=42,b=42} 42
1444000000000000// titi{a=42,b=42} 42