This message originated from Cribl Community Slack.
Click here to view the original link.
Hi, has anyone sent otel metrics from Cribl to Splunk? I'm currently sending the metrics into a metric index successfully.
But the histogram _buckets, _sum and _count are somehow not in the metrics in the index? Is the some sort of data engineering that needs to be done for Splunk to identify those bucket fields?
Data flow -> source otel(extract metrics) -> capture metrics in route -> apply pipeline to assign index based on conditions -> send to Splunk via HEC
As the data leaves Cribl the buckets, sum and count are present.
Solved
Cribl Stream: Sending OTel metrics to Splunk and histogram format
Best answer by Emily Ashley
Tbh, I know the OTel+Cribl side better than the Splunk side but this sounds like it may just be a format mismatch between how OTel represents histograms and how Splunk metric indexes expect to ingest them.
OTel histograms carry count, sum, bucket counts, bounds, and temporality together, while Splunk’s histogram docs describe separate
_bucket, _sum, and _count metrics.
If the target is specifically a Splunk metric index, I wonder whether the histogram needs to be emitted in Splunk’s histogram format before delivery, instead of relying on Splunk to infer it after indexing? If the receiving Splunk product supports OTLP more natively, it may also be worth trying Cribl’s OTLP Metrics function to rebatch/format the extracted metrics plus an OTel Destination instead of HEC. :thinking_face:
I’m not enough of a Splunk expert to say that with absolute confidence, but the public docs seem to point in that direction.
Tbh, I know the OTel+Cribl side better than the Splunk side but this sounds like it may just be a format mismatch between how OTel represents histograms and how Splunk metric indexes expect to ingest them.
OTel histograms carry count, sum, bucket counts, bounds, and temporality together, while Splunk’s histogram docs describe separate
_bucket, _sum, and _count metrics.
If the target is specifically a Splunk metric index, I wonder whether the histogram needs to be emitted in Splunk’s histogram format before delivery, instead of relying on Splunk to infer it after indexing? If the receiving Splunk product supports OTLP more natively, it may also be worth trying Cribl’s OTLP Metrics function to rebatch/format the extracted metrics plus an OTel Destination instead of HEC. :thinking_face:
I’m not enough of a Splunk expert to say that with absolute confidence, but the public docs seem to point in that direction.Here are the docs I was just checking out in case they’re useful:
- Splunk Platform: Use histogram metrics — help.splunk.com/en/splunk-cloud-platform/…/use-histogram-metrics
- OpenTelemetry Metrics Data Model — opentelemetry.io/docs/…/data-model
- Splunk Observability Cloud: Get histogram data in — help.splunk.com/en/splunk-observability-cloud/…/get-histogram-data-in
- Splunk Observability Cloud: Histogram metrics — help.splunk.com/en/splunk-observability-cloud/…/histogram-metrics
Sign up
Already have an account? Login
Login to the community
No account yet? Create an account
Using your Cribl Curious or University Account
User Login Employee loginEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.