Hello,
We are exploring running a self-hosted Cribl deployment on Kubernetes (EKS+Fargate) using the provided Cribl Helm charts. We've got a leader and worker pod running. We are now looking at persistent queuing.
My question is: what's the recommended configuration for ensuring no data loss in the case of a worker pod terminating with events still in the persistent queue?
We've been exploring the use of extraVolumeMounts and persistent volumes backed backed by EFS. We found this technique in this blog post (https://cribl.io/blog/persistent-queueing-using-the-logstream-workergroup-chart/) but we see it is not recommended in the the docs (https://docs.cribl.io/stream/deploy-kubernetes#using-persistent-storage-for-persistent-queueing).
We were wondering if there are better solutions that allow for a more automated recovery of events sitting in the persistent queue of the lost pod?