Skip to main content
Solved

Alert Management Strategies Needed For Search And Alerting Platform

  • February 7, 2026
  • 3 replies
  • 3 views
J
This message originated from Cribl Community Slack.
Click here to view the original link.

Has anyone got any strategies for managing alert suppression? Looking at using search as a complete search & alerting platform but worried about alert management

Best answer by cbreshears

So this was quite some time ago, and there might be WAY better ways to pull it off. I was playing with Kubernetes and testing if you could track when an unapproved pod became visible. Since I was getting a "snap shot" of the nodes any unapproved pod would throw an alert on every scheduled run. While this lookup was specific to the one alert, you could use a more generic lookup as long as you define what needs to be in the lookup to key off of. 
cribl dataset="K8_Metrics" namespace=='' image=='' image_id=='*' 
| dedup time_window=1000 by namespace, image, image_id 
| lookup matchType='first' output='pod_ID' ignorecase=false pos_pod_cmdb on namespace=namespace, image=image, image_id=image_id 
| lookup matchType='first' output='suppress_time' ignorecase=false POD_StateChange_Notifications on namespace=namespace, image=image, image_id=image_id
| extend is_approved = iif(pod_ID!=null, true, false)
| extend is_suppressed = iif(_time
The first lookup checks to see if the Pod details are in an "allowed" lookup. The second lookup checks to see if there is a suppressed item that matches in the lookup. If the pod is not approved and not in the suppress lookup, I write the event to the lookup and the alert is fired. (I realize now that I should have added a where earlier so I would not check suppression if pod was approved - but was testing, so sloppy.) I also had a scheduled search that would clean out any suppressions that were outside of the suppress bubble. 
cribl dataset="$vt_lookups" lookupFile="POD_StateChange_Notifications"
| extend is_suppressed = iif(_time
This is a pretty ugly example, but should get you started. Let me know if you have questions.

3 replies

cbreshears
  • cbreshears
  • Community Manager
  • February 7, 2026
Hey @user I haven’t had a chance to dig into this recently, but in the past I handled this with a simple state-tracking lookup tied to saved searches. The basic flow was: the search evaluates truthy → check a lookup to see if an alert was already triggered within the suppression window → if it’s not there, add/update the lookup with the suppression timestamp and set the alert field. If the alert is already in the lookup and still within the suppression window, don’t set the field that fires the alert. I have a FR open for notification suppression. That said, @user / @user might have better options for you.
Goat a load of this!
J
  • jp.lumsdaine917
  • Author
  • New Participant
  • February 7, 2026
Thanks @user you wouldn't happen to have some example KQL ? Do you mean you have a scheduled search to set the suppression status and then every individual scheduled search checks that table? Or do you have a single scheduled search responsible sending "alerts"
cbreshears
  • cbreshears
  • Community Manager
  • Answer
  • February 7, 2026
So this was quite some time ago, and there might be WAY better ways to pull it off. I was playing with Kubernetes and testing if you could track when an unapproved pod became visible. Since I was getting a "snap shot" of the nodes any unapproved pod would throw an alert on every scheduled run. While this lookup was specific to the one alert, you could use a more generic lookup as long as you define what needs to be in the lookup to key off of. 
cribl dataset="K8_Metrics" namespace=='' image=='' image_id=='*' 
| dedup time_window=1000 by namespace, image, image_id 
| lookup matchType='first' output='pod_ID' ignorecase=false pos_pod_cmdb on namespace=namespace, image=image, image_id=image_id 
| lookup matchType='first' output='suppress_time' ignorecase=false POD_StateChange_Notifications on namespace=namespace, image=image, image_id=image_id
| extend is_approved = iif(pod_ID!=null, true, false)
| extend is_suppressed = iif(_time
The first lookup checks to see if the Pod details are in an "allowed" lookup. The second lookup checks to see if there is a suppressed item that matches in the lookup. If the pod is not approved and not in the suppress lookup, I write the event to the lookup and the alert is fired. (I realize now that I should have added a where earlier so I would not check suppression if pod was approved - but was testing, so sloppy.) I also had a scheduled search that would clean out any suppressions that were outside of the suppress bubble. 
cribl dataset="$vt_lookups" lookupFile="POD_StateChange_Notifications"
| extend is_suppressed = iif(_time
This is a pretty ugly example, but should get you started. Let me know if you have questions.
Goat a load of this!
Terms & ConditionsAccessibility statement