Data getting dropped at cribl source

Hey there,

A few Cribl settings/behaviors could make big syslog events “disappear” compared to a direct path. Here are the main things I’ve heard to look at when troubleshooting this.
 

1. Check how you’re sending from the LB to Cribl (UDP vs TCP)

If your LB to Cribl leg is UDP, large events are the first suspect.

• Cribl’s Syslog Source supports UDP messages up to 16 KB. Anything bigger can get truncated or dropped before Cribl can really do anything with it.
• At higher volumes, UDP can also drop packets if buffers are too small.
• https://docs.cribl.io/stream/sources-syslog/

If these events are important, we’d recommend using TCP from the LB to Cribl for those feeds, especially the ones with large payloads.
 

2. Look at the Syslog Source stats in Cribl

Cribl tracks how many events a Syslog Source has received, buffered, and dropped.

• In the UI, go to:
  Monitoring → Data → Sources → <your syslog source> → Status
• Focus on the row for your syslog input and see if dropped is non‑zero during the time you’re testing.

If you see drops there, you know Cribl is dropping at the Source, not later.
 

3. Tune Source buffers if you see UDP drops

If you must stay on UDP:

• In the Syslog Source Advanced settings, increase the buffer size limit (events) from the default (e.g., bump it up and see if drops go away).
• Also look at the UDP socket buffer size (bytes) in the Source, and make sure the OS side (net.core.rmem_max) isn’t tiny. Small kernel buffers + high volume = dropped packets before Cribl even sees them.
   

4. Make sure the data is valid syslog for this Source

If the big events are slightly off‑spec, they may not parse cleanly:

• The Syslog Source expects RFC3164/5424 syslog. If the data is “syslog‑ish” but not actually valid, it can get flagged or broken weirdly.
• You can capture some events from that Source in Cribl and check for the internal field __syslogFail == true as a quick sanity check.

If the data isn’t really syslog, consider swapping to a Raw TCP/UDP Source with a custom event breaker instead of the Syslog Source.
 

5. Check Routes and Pipelines for explicit drops

If the events never show up when you search from the Cribl side, it’s worth ruling out a config issue:

• Routes: make sure there isn’t a Route filter that excludes those events and sends them to DevNull or no where.
• Pipelines: look for any Drop or Sample functions, or custom logic like “drop events when _raw.length is over X” or “drop certain hosts/sourcetypes.” Those often hit the “big” events first.

6. Destination backpressure behavior

Finally, if the data does get into Cribl but you don’t see it in your end destination

• Check the Destination’s backpressure behavior.
  If it’s set to Drop and the Destination is slow, Cribl will start dropping events at the Destination side instead of queueing them.

A lot of this is covered in https://docs.cribl.io/stream/sources-syslog/ so make sure to check that out!