Posting this to get community feedback and track our progress in case anyone encounters this issue in the future.

We are attempting to use the data cloning functionality on our F5 load balancer to duplicate data to our Cribl workers. We want to do this so that we can route, filter, and reduce production data in Cribl without impacting production operations. When the time comes, Cribl will be ready for the cutover without much impact at all, hopefully.

Right now, the F5 is sending data to our syslog servers, which then send to Splunk. The end goal is to replace syslog with Cribl, but we want to be able to see all the data in Cribl before then so that we can operate on it. We are successfully cloning data that comes from Splunk UFs and HFs, but the cloned syslog data from the F5 is not being seen by the Cribl worker.

The specific problem is this: The data is making it to the to the correct interface and port on the worker, but is not being picked up by Cribl for whatever reason.

I have verified that data is coming in by using tcpdump to view the packets. We have turned off the local firewall on the server. There are no iptables rules dropping the traffic. I have verified "net.ipv4.all.rp_filter" is not present in sysctl.

I'm not an F5 engineer, so I am mostly looking for feedback from that standpoint. Could the F5 be sending data in the wrong format, if that makes sense? My logic is that if the data is hitting the correct interface/port on the worker, then Cribl should be picking it up, but maybe that logic is flawed. Maybe there is a Linux configuration I am missing.

Funny thing is, we can see the UDP port monitoring traffic that the F5 sends to the worker, but not the data that it is cloning and sending.