After log configuration on port 1515 for some firewall logs, we observed logs are getting dropped and not all logs are forwarding to splunk.
We observed udp drops on RHEL OS level, where we increased the max_mem_buffer size to 256 MB , but still same thing -
Please check below output.
But still we are getting buffer issue as provided in below output.
[root@hostname ~]# cd /etc/sysctl.d/
[root@hostname /etc/sysctl.d]# ll
total 4
lrwxrwxrwx. 1 root root 14 Sep 12 16:14 99-sysctl.conf -> ../sysctl.conf
-rw-r-----. 1 root root 187 Dec 11 16:08 99-udp-tuning.conf
[root@hostname /etc/sysctl.d]# cat 99-udp-tuning.conf
net.core.rmem_max=268435456
net.core.wmem_max=268435456
net.core.rmem_default=268435456
net.core.wmem_default=268435456
net.ipv4.udp_rmem_min=262144
net.ipv4.udp_mem=65536 131072 262144
[root@hostname /etc/sysctl.d]#
[root@hostname /etc/sysctl.d]# netstat -us
IcmpMsg:
InType3: 141
InType8: 748254
InType13: 1
InType17: 3
OutType0: 748246
OutType3: 1321
Udp:
4544841348 packets received
56519 packets to unknown port received
1402052820 packet receive errors
659745 packets sent
1402052820 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InMcastPkts: 2
InBcastPkts: 40
InOctets: 5884308418800
OutOctets: 590438510065
InMcastOctets: 72
InBcastOctets: 9858
InNoECTPkts: 7689094316
InECT0Pkts: 514
MPTcpExt:
[root@hostname /etc/sysctl.d]#
→ Redhat Support suggested to check at application level - as they concluded - packets are getting dropped at application level.
Analyzing the provided sosreports I there are 2 udp sockets on port
1515 & 1514, I am not sure if you facing issue on the same app which
are listening on 1515 and 1514, Please confirm this. But I do see packet
drops on those 2 sockets.
Below is the analysis from the sosreport.
Below are the socket in which we see packets indicated in the last column.
> cat proc/net/udp | head -n 4
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
4398: 0100007F:0143 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 19243 2 ffff8d4a4e432800 0
5589: 00000000:05EA 00000000:0000 07 00000000:00000000 00:00000000 00000000 992 0 42210 2 ffff8d4a4766e900 1184
5590: 00000000:05EB 00000000:0000 07 00000000:00000000 00:00000000 00000000 992 0 42209 2 ffff8d4a47669900 647953554
The hexadecimal 05EA & 05EB becomes port 1514 & 1515.
Grepping the port 1514 and 1515 from the ss output I get below output.
In the below output if you see rb from skmem you will see its 33554432
& 2304 which means the read buffer of the socket is 33554432 & 2304.
> cat sos_commands/networking/ss_-peaonmi | grep -iE -A1 ":1514|:1515"
udp UNCONN 0 0 0.0.0.0:1514 0.0.0.0:* users:(("cribl",pid=456263,fd=25),("cribl",pid=456255,fd=26),("cribl",pid=456148,fd=26),("cribl",pid=456139,fd=26),("cribl",pid=456017,fd=29),("cribl",pid=456015,fd=26),("cribl",pid=2922,fd=33)) uid:992 ino:42210 sk:2168 cgroup:/system.slice/cribl.service <->
skmem:(r0,rb33554432,t0,tb212992,f4096,w0,o0,bl0,d1184)
udp UNCONN 0 0 0.0.0.0:1515 0.0.0.0:* users:(("cribl",pid=456263,fd=29),("cribl",pid=456255,fd=28),("cribl",pid=456148,fd=27),("cribl",pid=456139,fd=28),("cribl",pid=456017,fd=26),("cribl",pid=456015,fd=28),("cribl",pid=2922,fd=32)) uid:992 ino:42209 sk:2169 cgroup:/system.slice/cribl.service <->
skmem:(r0,rb2304,t0,tb212992,f4096,w0,o0,bl0,d647956376)
And the udp read buffer as per the OS is as below.
> cat sos_commands/kernel/sysctl_-a | grep -i core.rmem
net.core.rmem_default = 268435456
net.core.rmem_max = 268435456
NOTE:- So the buffer you are seeing in the udp socket is not taking from
the kernel setting but it seems the app has its own UDP buffer set from
the application code.
NOTE:- If you are concern of the same udp app which are listening on port
1514 & 1515 I'll suggest you to involve the app team and check the buffer
setting with them.We again made changes at UI -source side as below --
UDP socket buffer size (bytes) : 4294967295
Buffer size limit (events) : 16384
after committing - we observed , below changes are also not helping . packets are still getting dropped.
It would be great help if we get a solution on this !