In order to prevent uncontrolled data ingestion, Warp 10 comes with a throttling mechanism designed to limit both the number of Geo Time Series and the number of datapoints that an individual user can create.
Geo Time Series limits
The limit in the number of Geo Time Series (GTS) that a user can create or update is enforced on a monthly basis via a quota called
MADS for Monthly Active Data Streams. Every Geo Time Series which receives at least one update during the one month
period counts towards the
MADS quota. If a user attempts to update a GTS over his or her quota, an informational error message
will be returned stating that a particular GTS would exceed the defined quota for that user.
In a distributed Warp 10 deployment, the set of active data streams is synchronized between the hosts using a dedicated Kafka throttling topic.
The limit in datapoints is a daily one, conveniently called
DDP for Daily Data Points. But instead of enforcing a strict
limit on the number of datapoints uploaded daily, Warp 10 defines a rate of ingestion that each user cannot exceed. If
the amount of time to wait before a user can push data to stay within the defined rate goes above a certain limit, an
informational message will be returned to the calling user.
The use of a rate limit instead of a strict datapoint number limit was chosen to enable a user who has exceeded his or her
quota to still push datapoints but at a reduced rate. The caveat of this approach is that the rate should be changed
dynamically according to rules applying to the current daily amount of datapoints ingested. Fortunately the configuration
of the throttling mechanism allows that.
The default values for the
DDP limits and the maximum waiting time are set by the following configuration parameters:
``` throttling.manager.mads.default throttling.manager.rate.default throttling.manager.maxwait.default ```
The other throttling configuration parameters depend on the version of Warp 10 used.
Distributed Warp 10
ingress.kafka.throttling.brokerlist ingress.kafka.throttling.producer.clientid ingress.kafka.throttling.request.timeout.ms ingress.kafka.throttling.topic ingress.kafka.throttling.zkconnect ingress.kafka.throttling.consumer.clientid ingress.kafka.throttling.groupid ingress.kafka.throttling.consumer.auto.offset.reset throttling.manager.dir throttling.manager.period throttling.manager.rampup throttling.manager.estimator.cache.size throttling.manager.rate.default throttling.manager.mads.default throttling.manager.maxwait.default
Standalone Warp 10
throttling.manager.dir throttling.manager.period throttling.manager.rampup throttling.manager.rate.default throttling.manager.mads.default
throttling.manager.dir directory can receive so called throttling files which, if they end in
.throttle and have
not yet been read will be interpreted by the throttling manager to set custom limits for specific users (producers) or
This file is a simple text file whose lines contain the following fields
The per application limits cannot exceed the per producer limits.