Skip to main content
Solved

Cribl Stream: Persistent Queue Smart mode deprecation and performance

  • Nov 4, 2026 11:52 AM UTC
  • 1 reply
  • 0 views

This message originated from Cribl Community Slack.
Click here to view the original link.

Seeing a notice in the interface "Persistent Queue Smart mode is deprecated. No new development starting July 2026, followed by End of Support in January 2027 in which the feature will be removed and all deployments will automatically migrate to Always On mode, which uses a persistent queue buffer for all events." Switched this to 'Always' an the PQ levels just shot up, and began impacting performance. The mode 'Always' doesn't seem feasible.

Best answer by Brandon McCombs

Always On will indeed use more storage because at least some data will briefly be on disk for that input for any and all the processes that are receiving data for that input at any given time. For some customers CPU usage may increase as a result of Always On. For other customers, Smart mode caused more CPU overhead because of how often it was engaging/disengaging which caused additional context switching in Stream (and was the driver behind deprecating it). For those customers Always On was the better option and it turned out that was actually the case for many customers, which is why we began recommending Always On to many users for the last couple years before we officially decided to deprecate Smart mode. Both modes will use more CPU than not using sPQ at all since CPU time is required to interface with hardware. Total disk IO itself with Always On will be higher than with Smart because of the nature of how it works. Your environment may be in the camp where you could get by with using Smart mode without much additional visible CPU overhead or disk IO, possibly due to minimal backpressure downstream that would trigger it. For customers who have high frequency of backpressure from destinations (and the more destinations that exist the likelihood of this occurring increases) using Always On ends up being a net performance gain even with the added disk IO and inherent CPU usage required to interface with the hardware because Smart mode caused still higher CPU load. Some customers have hundreds of outputs so the likelihood of backpressure, even sporadically, at any given moment, is very high in those environments. They sized their environment accordingly to accommodate the higher disk IO that was incurred with Always On. For your env you may need more nodes to spread out the disk IO load. You could scale up but the new processes would add to the contention on the existing disks so adding more nodes will add more disks. For comparison, in Cloud we've only ever used Always On mode for sPQ.
View original

1 reply

Always On will indeed use more storage because at least some data will briefly be on disk for that input for any and all the processes that are receiving data for that input at any given time. For some customers CPU usage may increase as a result of Always On. For other customers, Smart mode caused more CPU overhead because of how often it was engaging/disengaging which caused additional context switching in Stream (and was the driver behind deprecating it). For those customers Always On was the better option and it turned out that was actually the case for many customers, which is why we began recommending Always On to many users for the last couple years before we officially decided to deprecate Smart mode. Both modes will use more CPU than not using sPQ at all since CPU time is required to interface with hardware. Total disk IO itself with Always On will be higher than with Smart because of the nature of how it works. Your environment may be in the camp where you could get by with using Smart mode without much additional visible CPU overhead or disk IO, possibly due to minimal backpressure downstream that would trigger it. For customers who have high frequency of backpressure from destinations (and the more destinations that exist the likelihood of this occurring increases) using Always On ends up being a net performance gain even with the added disk IO and inherent CPU usage required to interface with the hardware because Smart mode caused still higher CPU load. Some customers have hundreds of outputs so the likelihood of backpressure, even sporadically, at any given moment, is very high in those environments. They sized their environment accordingly to accommodate the higher disk IO that was incurred with Always On. For your env you may need more nodes to spread out the disk IO load. You could scale up but the new processes would add to the contention on the existing disks so adding more nodes will add more disks. For comparison, in Cloud we've only ever used Always On mode for sPQ.