Loading...
 

SysAdmin Blog

syslog-ng and RcvbufErrors on Linux

Alexander Bochmann Tuesday 10 of May, 2016
We're running a syslog-ng installation to collect syslog data from quite a lot of systems (and then selectively feed them into our Splunk installation). Almost all of these send syslog via UDP.

Recently, when adding a couple more machines, I noticed that the syslog server is dropping UDP datagrams:

udp RcvbufErrors
# netstat -su | grep -A6 "^Udp:"
Udp:
    518026364 packets received
    36078 packets to unknown port received.
    23164168 packet receive errors
    1248583 packets sent
    RcvbufErrors: 23164167
UdpLite:


Yikes!

This is mentioned in the syslog-ng OSE docs, but it seems no one here ever got to that section, including myself.

So, in that context I learned about the so-rcvbuf() parameter to the udp() source in syslog-ng, and the Linux kernel net.core.rmem_max sysctl...

Kernel configuration
# sysctl -w net.core.rmem_max=16777216

(add the same parameter to /etc/sysctl.conf)

syslog-ng.conf
source s_net {  
                udp(ip(0.0.0.0) port(514) so-rcvbuf(8388608)); 
};

(There's no reason why so-rcvbuf() couldn't be the same as rmem_max, and neither needs to be a multiple of 1024 - both just bad habits of mine...)

Don't increase net.core.rmem_default, as that would make the Linux kernel use a bigger buffer for every UDP socket being created on the system.

The RcvbufErrors counter hasn't been increasing since that change, but I'll add monitoring for that, so drops won't go unnoticed in the future.