Cribl is the Cherry On the Top of the Splunk Sundae!

someone turned on advertisement mode on <@U01C35EMQ01&gt;

but as I said... usually most fields are extracted at search time, and index time processing is rather limited. also, I think you'll get a handle on how to properly use Cribl very quickly

You can get metrics events to work with Splunk IT Essentials and publish metrics function in Stream, bit fiddly, but I have done it. Unfortunately, customer was in a closed environment, so could not export pack.I echo what the esteemed community brethren have stated above. Most Splunk parsing is done at search time > as it makes sense for schema-on-fly. However, there are some example where this does not work great. Lookups are a key one - I much prefer managing these within Stream.

But do not use the "hack" - things blow up and it messes with Splunk processing queues and you can bjork the metrics components, so you cannot even troubleshoot or fix. Use `raw` HEC endpoint, if you want Splunk to do some index-time parsing. Personally, I try best to use push-based with Splunk as a destination with HEC over SplunkTCP. Better on your network and the perf has dramtically improved from early days.

<@U03FQSY3JCF&gt; i had similar issue with my udp & perfmon metrics data using passthru to Splunk Cloud after upgrading to Cribl Stream 4.0.0 and updating our Splunk Cloud destination's max s2s version to use v4 instead of v3. My currently working solution to continue getting my metric data through Cribl Stream into our Splunk Cloud indexers was to clone my Splunk Cloud destination to another one and set s2s to use v3 instead of v4, filter these events out with a new Route using passthru pipeline and output that route to the cloned v3 Splunk Cloud destination.We've submitted a support case for this, i've recreated this by also sending the same metrics using passthru with s2s v3 to a Splunk Enterprise dev instance and also fails to get there when i flip s2s to v4

would one of you mind sending me what the HEC metrics format for PerfmonMetrics should look like to properly send to splunk cloud's hec endpoint? I'm trying this but it's not working:```{ "source": "Perfmon:Network", "host": "W2022GOLDTEST5", "sourcetype": "PerfmonMetrics:Network", "index": "win_em_metrics", "entity_type": "Windows_Host", "time": 1674495318, "event": "metric", "fields": { "_value": "23888.970133604955", "metric_name": "Network Interface.Bytes Total/sec", "collection": "Network", "instance": "vmxnet3 Ethernet Adapter" }, "cribl_pipe": "perfmon_metrics"}```

Are you using the <https://docs.cribl.io/stream/publish-metrics-function/|Publish Metrics> function to ensure all you dims and values are correct, before sending to HEC destination?

No, i thought if I created the fields that needed to be in the event and removed _raw then it would already be in the right format. is that not accurate? I haven't tried the Publish metrics function yet but can give it shot.

That is the most sure fire way I have got Splunk to play nice with metrics. Takes some tinkering to ensure you get the dims etc in right format.

Not that I can see, but you want it to look similar to the below example, before it leaves Stream.

ok I'll take a look and see if I can make some progress. are there any glaring issues you can see on the format I pasted above from a field perspective?

isn't that format for log to metrics vs what perfmon metrics should look like?

Nope, that is what the Publish Metrics function will give you, and ensure it is sent as a metrics, rather than event.

any chance you can paste your json of the pipeline you use for your perfmon metrics from cribl?

The ones I have done are locked in a secure customer env - sorry. If you can give me some redacted ones - I could try and replicate.

<@U03FQSY3JCF&gt; - check this thread out - https://cribl-community.slack.com/archives/CPYBPK65V/p1653030903605899