Is there a pattern that I can use to suppress Splunk logs based on data that is part of the _raw field? So far I have come up with a pipeline that uses Regex Extract, Suppress, and Eval function. I am extracting the key:value pair that I need from the _raw field, utilize it in my suppression function, and then remove it from the logs before sending the event to Splunk. While this seems to be working, I wonder if there is a better way to achieve this especially if we want to add additional key:value pairs to suppress on in the future? I was looking into the Parser function that allows me to extract all key:value pairs from the _raw field but I could not find a way to remove all those from the event before sending it to Splunk.
Solved
Patterns for suppressing Splunk cooked data
Best answer by Jon Rust
Hold on, i misread your post
Use the Parser function to Extract _raw into a new field, maybe __myfield. Now you can refer to __myfield in your suppression action.
Can you supply a sample log? Absolutely we can add/remove/modify fields in any part of the event. Id just like to see real world example before getting too deep into particulars.
Here is an example log that I see in Cribl. Just FYI, I replaced the values for some of the fields because that info is internal to the company. So based on the below example, I am trying to find a way to be able to suppress on any of the key:value pairs that are part of the _raw field.
{ "_raw": "{\"Timestamp\":\"2022-05-03T14:51:25.0606688+00:00\",\"Level\":\"Information\",\"MessageTemplate\":\"Successfully retrieved data for {service}\",\"RenderedMessage\":\"Successfully retrieved data for MyService\",\"Properties\":{\"count\":0,\"WorkspaceID\":1234567,\"SourceContext\":\"MyService.Worker.DataRetriever\",\"Priority\":2,\"InstanceIdentifier\":\"c783c624-d5c5-4eb7-a08c-9f50634b08e4\",\"DocumentID\":1234567,\"CorrelationID\":\"ff36e36b-ce29-4cad-9483-c2864dbe84aa\",\"HostName\":\"my-host\",\"ImageTag\":\"v1.0.290\",\"System\":\"MyService\",\"ProcessID\":19056,\"Application\":\"2.224.0\",\"SubSystem\":\"Worker\"}}", "_time": 1651589485.060668, "source": "2.224.0", "host": "10.10.1-.100", "sourcetype": "MyService", "index": "myIndex", "timestartpos": "14", "timeendpos": "47", "date_second": "25", "date_hour": "14", "date_minute": "51", "date_year": "2022", "date_month": "may", "date_mday": "3", "date_wday": "tuesday", "date_zone": "0", "punct": "{\"\":\"--::.+:\",\"\":\"\",\"\":\"__{}____{},__{}\",\"\":\"_____", "MessageTemplate": "\", "cribl_pipe": "suppression-test" }Hold on, i misread your post
Use the Parser function to Extract _raw into a new field, maybe __myfield. Now you can refer to __myfield in your suppression action.
I was just about to post that we found out that we can do what you are suggesting Jon. I think that does what we need. Thanks for the help Jon!
Excellent! The trick is making sure your JSON objects are parsed before you start trying to manipulate the contents. In the preview window, if they have an a next to them, it means theyre a string. Only string functions will work. But if they have a {} next to them, its seen as an object and ready to be pulled apart like a lego kit. You can use the Parser function, or JSON.parse() in an Eval to parse a string that is proper JSON into an object.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.