Throttling

In order to prevent uncontrolled data ingestion, Warp 10 comes with a throttling mechanism designed to limit both the number of Geo Time Series and the number of datapoints that an individual user can create.

Geo Time Series limits

The limit in the number of Geo Time Series (GTS) that a user can create or update is enforced on a monthly basis via a quota called MADS for Monthly Active Data Streams. Every Geo Time Series which receives at least one update during the one month period counts towards the MADS quota. If a user attempts to update a GTS over his or her quota, an informational error message will be returned stating that a particular GTS would exceed the defined quota for that user.

In a distributed Warp 10 deployment, the set of active data streams is synchronized between the hosts using a dedicated Kafka throttling topic.

Datapoints limits

The limit in datapoints is a daily one, conveniently called DDP for Daily Data Points. But instead of enforcing a strict limit on the number of datapoints uploaded daily, Warp 10 defines a rate of ingestion that each user cannot exceed. If the amount of time to wait before a user can push data to stay within the defined rate goes above a certain limit, an informational message will be returned to the calling user.

The use of a rate limit instead of a strict datapoint number limit was chosen to enable a user who has exceeded his or her DDP quota to still push datapoints but at a reduced rate. The caveat of this approach is that the rate should be changed dynamically according to rules applying to the current daily amount of datapoints ingested. Fortunately the configuration of the throttling mechanism allows that.

Configuring throttling

Default values

The default values for the MADS and DDP limits and the maximum waiting time are set by the following configuration parameters:

```
throttling.manager.mads.default
throttling.manager.rate.default
throttling.manager.maxwait.default
```

The other throttling configuration parameters depend on the version of Warp 10 used.

Distributed Warp 10

ingress.kafka.throttling.brokerlist
ingress.kafka.throttling.producer.clientid
ingress.kafka.throttling.request.timeout.ms
ingress.kafka.throttling.topic
ingress.kafka.throttling.zkconnect
ingress.kafka.throttling.consumer.clientid
ingress.kafka.throttling.groupid
ingress.kafka.throttling.consumer.auto.offset.reset

throttling.manager.dir
throttling.manager.period

throttling.manager.rampup
throttling.manager.estimator.cache.size
throttling.manager.rate.default
throttling.manager.mads.default
throttling.manager.maxwait.default

Standalone Warp 10

throttling.manager.dir
throttling.manager.period
throttling.manager.rampup
throttling.manager.rate.default
throttling.manager.mads.default

Throttling file

The throttling.manager.dir directory can receive so called throttling files which, if they end in .throttle and have not yet been read will be interpreted by the throttling manager to set custom limits for specific users (producers) or applications.

This file is a simple text file whose lines contain the following fields

PRODUCER_UUID:MADS:RATE::#
+APPLICATION_NAME:MADS:RATE::#

The per application limits cannot exceed the per producer limits.