greedy zabbix_sender chokes on too many data

Greedy zabbix_sender chokes on too many data

zabbix_sender tool performs a partial database import when fed with too many data at once.

I am using zabbix_sender tool to inject into a zabbix database data samples related to computer network statistics.

I start with a clean zabbix database, after clearing zabbix historical data for some items, so that the database is empty. Then I load new data with zabbix sender tool, and then I analyze the data mostly in graphical mode, checking the behaviours of the systems under particular loads, using zabbix graphical interface for historical navigations, statistics, and graph overlapping. When I am done with the analysis, I again delete historical values, and load a new batch of values.


I need to manage about 500.000 records. The problem I am describing here was not evident until data processed by my scripts was below 40.000 records. Problem was not evident using non differential valued zabbix items.


Problem is that, when loading the data via zabbix_sender, from an input file 

[...]
100.10.4.8 FailedConnectionAttempts 1434719102 114715
100.10.4.8 ResetConnections 1434719102 51829088
100.10.4.8 CurrentConnections 1434719102 298
100.10.4.8 SegmentsReceived 1434719102 321966025
100.10.4.8 SegmentsSent 1434719102 1445309521
100.10.4.8 SegmentsRetransmitted 1434719102 64705214
100.10.4.8 ActiveOpens 1434719402 981491
100.10.4.8 PassiveOpens 1434719402 312320429
100.10.4.8 FailedConnectionAttempts 1434719402 114732
100.10.4.8 ResetConnections 1434719402 51833025
100.10.4.8 CurrentConnections 1434719402 459
[...]

Using the following command line:
zabbix_sender -z 127.0.0.1 -T -i zbx-inputfile.txt

I get no errors when executing this, but older data are badly imported.

I think this is depending on data being processed by different threads of zabbix server (zabbix trappers) which can not talk each other considering that some data is to be interpreted as difference with the previous value. Differential data requires all the values to be sequentially injected and processed by a single zabbix-trapper thread.
Strangely, I did not find any mention to this potential problem, but you can not expect anything good if you are going to insert historical values in any series which is to be considered in differential mode, because data is processed before being inserted in the database, and the difference with the previous value is the actual value which goes in the database.
I tried to use -r (realtime) option in the zabbix_sender command line, but this did not give any improvement (actually I did not understand precisely what the realtime option is for).

I know that I am using Zabbix and zabbix_sender tools in a way that is probably quite far from the designer scopes, but I love this great software.

I was able to solve the issue slowing the speed at which data are sent to zabbix_sender.
In order to do this I built a filter tool which slows down a unix communication pipe.

The tool is a simple perl script called slower-pipe which sends some lines, then wait a specific time before sending the next chunk

cat zbx-inputfile.txt | slower-pipe.pl 100 2000 | zabbix_sender -z 127.0.0.1 -T -vv -i -

Using the tool to slow the input feed (100 lines per chunk and 2 seconds wait) the problem vanished.
The reason for this kludge to be effective lies probably in the fact that slowing down the input data flow, a single zabbix-trapper thread is considered enough, and calculations of the differential items are correctly processed.

here is the (perl) source code for slower-pipe.pl
just copy it, save it into slower-pipe.pl and then chmod +x appropriately

#!/usr/bin/perl
#
# mgua@tomware.it
#  this code is free software, GPL licensed.
#
# produces a slower passage over a pipe
#
# Syntax:
# slower-pipe chunksize msec
#   parameter1: chunksize (lines)
#   parameter2: milliseconds to wait after each chunk
#
#

use Time::HiRes qw(usleep nanosleep); # microsecond sleep

my $DEBUG = 0;
my $chunksize = 1;
my $msecwait = 10;      # milliseconds wait

if (($#ARGV + 1) ne 2) {
        die("Usage: slower-pipe chunksize msecwait\n   chunksize: number of lines per block\n   msecwait: milliseconds to wait after sending each block\n");
}

if ($ARGV[0] ne "") {
        $chunksizein = $ARGV[0];
        if ($chunksizein > 1) { $chunksize = $chunksizein; }
        print("chunksizein = [$chunksizein] chunksize=[$chunksize]\n") if $DEBUG;
}

if ($ARGV[1] ne "") {
        $msecwaitin = $ARGV[1];
        if ($msecwaitin > 1) { $msecwait = $msecwaitin; }
        print("msecwaitin = [$msecwaitin] msecwait=[$msecwait]\n") if $DEBUG;
}

my $usecwait = 1000 * $msecwait;


while () {
        print($_);
        if (($. % $chunksize) == 0) {
                print(" delay\n") if $DEBUG;
                usleep($usecwait);
        }
}

enjoy!


@mgua