Skip to main content
Question

How can I troubleshoot a cribl destination (Splunk HEC) not sending data?

  • March 11, 2025
  • 11 replies
  • 77 views

Hello.How can I troubleshoot a cribl destination (Splunk HEC) not sending data? destination is seen live/green check, but no data flowing per tcpdump on source or destination. this goes out through a router and same data arriving in the main index correctly but want in a separate one. curl from cribl to splunk works (`curl -vk https://172.16.x.y:8088/services/collector/event -H 'Authorization: Splunk TOKEN' -d '{"event":"test"}'`)Thanks

11 replies

Jon Rust
Forum|alt.badge.img
  • Employee
  • March 11, 2025

my first step would be to send to `nc` running on a local host and interrogate the actual payload


  • Author
  • Employee
  • March 11, 2025

Trying that but I suspect that I will get nothing. Destination out traffic charts are at 0.Not sure if rules order matter in router destination but tried to change those as I have few disabled one/set false.


  • Author
  • Employee
  • March 11, 2025

For now, not getting expected data. If using destination Test option, I get it in tcpdump and splunk. I would think the problem is likely not in destination but before. But not sure why as the main routing is working and not this one.


Jon Rust
Forum|alt.badge.img
  • Employee
  • March 11, 2025

Running the test from within the HEC destination doesn't involve routing in any way


  • Author
  • Employee
  • March 11, 2025

yes and this part is working. so that means destination is configured fine, right? in this case, problem is earlier


  • Author
  • Employee
  • March 11, 2025

strangely after switching back localhost to splunk system, now, I have got data in tcdpump 8088but strange correct/incorrect marking...all in aws network


  • Author
  • Employee
  • March 11, 2025

but no data in splunk UI


Jon Rust
Forum|alt.badge.img
  • Employee
  • March 11, 2025

the test function should produce data in splunk, if you specify the correct index


Jon Rust
Forum|alt.badge.img
  • Employee
  • March 11, 2025

fix that first. then tackle routing config


  • Author
  • Employee
  • March 11, 2025

Ok. got it. that was the router "final" option check. now got it in splunk


  • Author
  • Employee
  • March 11, 2025

Thanks