Are source persistent queue events lost when PQ is disabled in Cribl Stream?

Question

The Cribl Stream documentation states that disabling source persistent queue (sPQ) or shutting down the source before its queues drain will "orphan any events still queued on disk." Does this mean the events are permanently lost?

Environment

• Cribl Stream (on-prem and Cloud)
• Source Persistent Queues (sPQ)

Answer

Events in source persistent queue are not permanently lost when PQ is disabled. The term "orphan" in this context means the events are left in place on disk and not deleted by the application. However, recovery depends on your storage configuration and deployment type.

Behavior differs by storage type and deployment:

1. Local storage (on-prem or Cloud): Orphaned PQ data is tied to the specific worker node where it was written. When sPQ is re-enabled for the same source on the same worker, the worker process will begin to flush the queued events. The events remain on disk indefinitely until sPQ is re-enabled. PQ data on local storage cannot be recovered by a different worker node.
2. Shared storage (NFS or S3): PQ data persists in the shared storage bucket or mount. If the original worker node is lost or decommissioned, surviving worker nodes can potentially recover the data through the orphan management and PQ revival mechanism (pqWorker). However, this recovery process is designed for scenarios where a worker node has died or been removed — it does not automatically drain PQ that was administratively disabled on a live source. You should still re-enable sPQ on the same live worker to drain the queue.

Important caveats:

• For local storage, the data must remain on the same worker node. External factors such as Kubernetes pod rescheduling can cause data to be truly lost if the underlying local storage is destroyed.
• The pqWorker process (available in Cloud and on-prem with shared storage starting in 4.18.0) is specifically designed to drain orphan PQ directories from dead or terminated worker nodes. It does not automatically drain PQ files when PQ is simply disabled on a live source.
• There is currently no built-in "drain before disable" mechanism.

Additional Information

For more information, see Cribl Stream - Source Persistent Queues documentation.