Loading

  1. Configuration
  2. Handler documentation and Hello World

TCP Text Plugin

HTTP Plugin is build on top of HTTP protocol. TCP Text is a low layer plugin, designed for really custom protocols.

For better performances, we strongly advise you to use SHM extension to build buffers for your incoming data over TCP. You must not do an UPDATE on each line of text you receive!

The idea behind TCP plugin is the same than HTTP plugin: Trigger WarpScript on receive event. For TCP plugin, the event is a line break \n in the stream. We assume the text is encoded in UTF-8.

This plugin was designed for high rate data processing with a complex GTS model in memory...

Configuration

Your custom configurations can be stored in a new etc/conf.d/99-myWarp10.conf file. Your custom configurations will override all the predefined Warp 10 configuration.

// Activate the TCP plugin
warp10.plugin.tcp = io.warp10.plugins.tcp.TCPWarp10Plugin
// Directory where TCP 'spec' files will be located
tcp.dir = ${standalone.home}/TCP_plugin
// Period (in ms) at which to rescan 'tcp.dir' for spec files changes (defaults to 60s)
tcp.scanperiod = 5000

Create /opt/warp10/TCP_plugin, and make sure warp10 user has the rights to read it. You can create sub directories if you have several applications, the plugin will crawl them too.

Restart your Warp 10 instance to activate the plugin.

Handler documentation and Hello World

To run test below, you need to activate the Debug Extension to write in Warp 10 logs.

For each port you want to listen on, create a new WarpScript file in your /opt/warp10/TCP_plugin. It will contain the queue configuration and the WarpScript macro to execute when a new line is received.

For example, create port43211.mc2, and make sure warp10 user can read it :

{
  'mode' 'server'
  'host' '0.0.0.0'
  'port' 43211
  'timeout' 5000    // Optional, ms, defaults to 0 (no timeout)
                    //  when timeout is reached, the macro will be called anyway, 
                    //  with NULL on top of the stack if nothing was received in the meantime
  'parallelism' 4   // Optional, defaults to 1
  'qsize' 2048      // Optional, defaults to 1024
  'partitioner'     // Optional, the macro must return a LONG which will be a queue number.
  <% 
    // on the stack, you will find the packet content : 
    //  from bottom to top: IP (STRING), Source port (LONG), and the text line (STRING) 
    // you may partition up to content received.
    SNAPSHOT 'textLine' STORE
    'partitionner input stack snapshot=' $textLine  + STDOUT
    RAND 4 * TOLONG //return a random number.  
  %> 
  'maxClients' 10    // Optional, defaults to 1
  'maxMessages' 1024 // Optional, defaults to 1024
  'macro' <%  // the macro executed on new lines received 
    // on top of the stack, there is a list of packet.
    'packetLIST' STORE
    <% $packetLIST ISNULL %>
    <%
      'timeout reached, nothing received' STDOUT
    %>
    <%
      $packetLIST
      <%
        'packet' STORE
        //each packet is a list of three item: IP (STRING), Source port (LONG), and the text line (STRING)
        'packet from IP:' $packet 0 GET + ' port:' + $packet 1 GET TOSTRING + 
        ' content=' + $packet 2 GET + 
        STDOUT
      %> FOREACH
    %> IFTE
  %>
}


Wait for the refresh time (tcp.scanperiod), then check that the 43211 port is opened on your system:

nmap -p 43211 localhost

Then, tail the warp10.log to display STDOUT in real time:

tail -F /opt/warp10/logs/warp10.log

You should see 4 timeout messages every 5second (parallelism = 4, timeout = 5000).

In another terminal, send data to the opened port:

printf 'Hello Word!\nThis is a TCP client!\n\n' | nc -N localhost 43211

You should see in the log one message per line of text (even for the empty one at the end):

timeout reached, nothing received
partitionner input stack snapshot='127.0.0.1' 57096 'Hello%20Word%21' 
partitionner input stack snapshot='127.0.0.1' 57096 'This%20is%20a%20TCP%20client%21' 
partitionner input stack snapshot='127.0.0.1' 57096 '' 
packet from IP:127.0.0.1 port:57096 content=Hello Word!
packet from IP:127.0.0.1 port:57096 content=This is a TCP client!
packet from IP:127.0.0.1 port:57096 content=
timeout reached, nothing received
timeout reached, nothing received

To avoid blowing up your log file, don't forget to comment out the timeout STDOUT every five second!

Note you can force the origin port with the flag -p, for example printf 'Hello' | nc -N localhost 43211 -p 42000, but you have to wait for the system to close the TCP port. The TCP Text plugin will never close the port. So it will remain on TIME_WAIT (during 1 minute on most linux system).

The parallelism level is the number of worker thread. The partitionner can add several queues to distribute the workload differently, up to line content. The schematic below explains the configuration:

TCP config explained