How are you handling token persistence?

<@U02QJ374Z3R&gt; there are many who are using Splunk LB, but as far as I know .. using Indexer discovery + updating the Indexer discovery often .. I am not sure how often most customers updating the token ..

Once you update the Token, Cribl will not impact data delivery since .." Worker Process Rolling RestartDuring a restart, to minimize ingestion disruption and increase availability of network ports, Worker Processes on a Worker Node are restarted in a rolling fashion. 20% of running processes – with a minimum of one process – are restarted at a time. A Worker Process must come up and report as started before the next one is restarted. This rolling restart continues until all processes have restarted. If a Worker Process fails to restart, configurations will be rolled back. "

with Cribl there are 2 ways (that I can think-off) to update the Indexer Discovery token. Manually in the UI or Cribl API

I don't use Indexer Discovery, nor do I recommend it to clients. Just say no.

<@U01J549PR6Y&gt; - it's not that I want to update the token regularly in Cribl (if at all unless a breach or something). But when I recycle my cluster manager the auth tokens get wiped on the Splunk side and would therefore be invalidated in Cribl indexer-discovery.. Granted this is more of a Splunk question, but figured enough Cribl customers leverage Splunk LB destination that someone has a solution for it.

<@UEGNG8MJB&gt; any particular reason? Curious what better method exists to maintain a working list of online indexers.

i used DNS. one name, all the indexer IPs behind it.I'm curious why a CM restart trashes the token. Never seen that before.

Not a restart, but swapping out the EC2 instance for updated AMI etc.

We also recycle the indexers for the same reasons. (security compliance)

As Jon said, most common that I have seen are the DNS alternative

DNS. Cribl handles DNS round robin waaaay better than Splunk. And being an old-school network guy who prefers an NLB over any round robin DNS, that says something. The programmers did an excellent job of leveraging DNS entries that are loaded with IPs. And the Cribl load balance approach is better than Splunk's.

I've found DNS is just reliable with Stream.

Thanks. So then the 'discovery' config would look something like this?

correct!

great....stay tuned.

Worked like a charm. Thanks for the guidance on this, much appreciated.

One follow up, which backpressure method best suits Splunk LB destination?The way I understand this setting, is that if the destination (ie Splunk indexers) cannot receive data:» block - buffer in memory on cribl workers » drop - /dev/null it » PQ - queue it on disk on the cribl workers until destination is accepting data

enable PQ

and set-up some constraints around the storage

i would enable compression too (not on by default iirc)

Reading https://docs.cribl.io/stream/persistent-queues/#persistent-queue-details-and-constraints|this, it looks like the PQ's use workers' storage. Hard to but a number on the amount of storage a worker can allocate for PQ, since it's relative to the amount of data you'd be sending to the destination, right?

if you were doing 240 GB per day, that's 10 GB per hourdivided by 2 workers == 5 GB per worker; then factor in compression

yesrate of sending * expected down time / number of workers * compression expected

and add 50% :slightly_smiling_face:

napkin math ftwthanks a lot Jon. I did set it to 5GB, since our workers are on Fargate and have 20GB of ephemeral storage by default.And since PQ is worst case sort of scenario, ie the entire indexer cluster is :dumpsterfire: , seems to be a safe setting.