We're running a syslog-ng installation to collect syslog data from quite a lot of systems (and then selectively feed them into our Splunk installation). Almost all of these send syslog via UDP.
Recently, when adding a couple more machines, I noticed that the syslog server is dropping UDP datagrams:
Yikes!
This is mentioned in the syslog-ng OSE docs, but it seems no one here ever got to that section, including myself.
So, in that context I learned about the so-rcvbuf() parameter to the udp() source in syslog-ng, and the Linux kernel net.core.rmem_max sysctl...
(add the same parameter to /etc/sysctl.conf)
(There's no reason why so-rcvbuf() couldn't be the same as rmem_max, and neither needs to be a multiple of 1024 - both just bad habits of mine...)
Don't increase net.core.rmem_default, as that would make the Linux kernel use a bigger buffer for every UDP socket being created on the system.
The RcvbufErrors counter hasn't been increasing since that change, but I'll add monitoring for that, so drops won't go unnoticed in the future.
Recently, when adding a couple more machines, I noticed that the syslog server is dropping UDP datagrams:
udp RcvbufErrors
Copy to clipboard
# netstat -su | grep -A6 "^Udp:" Udp: 518026364 packets received 36078 packets to unknown port received. 23164168 packet receive errors 1248583 packets sent RcvbufErrors: 23164167 UdpLite:
Yikes!
This is mentioned in the syslog-ng OSE docs, but it seems no one here ever got to that section, including myself.
So, in that context I learned about the so-rcvbuf() parameter to the udp() source in syslog-ng, and the Linux kernel net.core.rmem_max sysctl...
Kernel configuration
Copy to clipboard
# sysctl -w net.core.rmem_max=16777216
(add the same parameter to /etc/sysctl.conf)
syslog-ng.conf
Copy to clipboard
source s_net { udp(ip(0.0.0.0) port(514) so-rcvbuf(8388608)); };
(There's no reason why so-rcvbuf() couldn't be the same as rmem_max, and neither needs to be a multiple of 1024 - both just bad habits of mine...)
Don't increase net.core.rmem_default, as that would make the Linux kernel use a bigger buffer for every UDP socket being created on the system.
The RcvbufErrors counter hasn't been increasing since that change, but I'll add monitoring for that, so drops won't go unnoticed in the future.