Recently active topics

Known issue page(s) updated: CRIBL-45369 | Worker Process memory leaks while Collection jobs run

QuestionWhy does Members & Teams show the Admin role for users that I've granted the Owner role via group membership in my SSO IDP?Environment Cribl environment with SSO configured for login AnswerFor SSO users, the role in the Cribl UI is updated when a user signs in and it reflects the role granted the last time they signed in. Therefore, the Cribl UI may not always align with the SSO IDP. The IDP is the source of truth for current permissions for SSO users.

ObjectiveWrite an initial Getting Data In (GDI) datatype rule filter in Cribl Search Lakehouse using the correct available fields.EnvironmentCribl Search 4.17.0 (Lakehouse engines) Search > Data > Getting Data In Initial Datatype / Dataset rule KQL expression fieldProcedure Identify which fields are available at the initial GDI rule stage. Only a small set of metadata exists at this point, before any parsing or time extraction has occurred: _raw -- full raw event string as received from the source __inputId -- ID or name of the GDI source that ingested the event port -- destination TCP/UDP port of the source host -- source host (IP or FQDN) that sent the data Write the filter expression using only the fields listed above. Do not reference fields that are created after parsing or later in the pipeline, such as type, action, source_ip, dest_ip, datatype, _time, or schema. Conditions that reference those fields will not match because they do not yet exist when t

New known issue page(s) added: CRIBL-45369 | Worker Process memory leaks while Collection jobs run

SymptomWhen a destination, pipeline, or route is deleted from within a pack in the Cribl Stream UI, the resource appears to be removed. However, after a Commit & Deploy the resource reappears. Exporting the pack with local changes and re-importing it also brings the deleted resource back.Additional observed symptoms:Copying the pack to another worker group with Overwrite enabled does not remove the deleted resource in the destination group. The deletion does not appear in the commit diff.EnvironmentCribl Stream (all versions) Deployment type: Cloud (managed) and self-hosted Component: PacksResolutionExport the pack in merge mode, remove the resource from the exported file, bump the pack version, and re-import with Overwrite. Merge mode collapses the default/ and local/ layers into a single flat default/ layer. Because the re-imported pack contains no default entry for the deleted resource, there is nothing left to merge back. In the Cribl UI navigate to Processing → Packs, find the

SymptomCribl Stream Worker pods deployed in Kubernetes show unexpectedly high CPU and memory usage even though processing and log-ingestion volume is low. One or more pods may later restart with OOMKilled.The two symptoms can have different causes. CPU usage can come from too many Worker Processes or from work pinned to one Worker. The OOMKill can come from memory used by Cribl, or by an external process that Cribl started and that is not represented in Cribl Worker Process metrics.kubectl describe pod may report:Reason: OOMKilled Exit code 137 or an equivalent Kubernetes out-of-memory termination messageEnvironmentCribl Stream Worker Group deployed in Kubernetes. Worker pods have Kubernetes CPU and memory requests or limits configured.Resolution Capture both symptoms from the affected pod. Run: kubectl describe pod <pod-name> -n <namespace> kubectl top pod <pod-name> -n <namespace> --containers Record the pod name, OOMKill timestamp, restart count, container re

SymptomAfter a Cribl Stream Leader restart or configuration change, port 4200 is not listening and Workers or Edge Nodes cannot reconnect to the Leader.The Leader logs can include:Shutdown:CB:Failed name: SocketRouter Error: Cannot read properties of undefined (reading '_connections')The logs can also show RpcNoWorkersRegisteredError: No workers registered or out-of-memory errors in $CRIBL_HOME/log/cribl_stderr.log.EnvironmentCribl Stream in a self-hosted distributed deployment A Cribl Stream Leader managing many Worker Groups, Fleet mappings, or Edge Nodes Leader memory that is insufficient for the resource demand during a cold startResolutionVerify whether the Leader is listening on port 4200:ss -tulpn | grep 4200 Review $CRIBL_HOME/log/cribl_stderr.log for out-of-memory errors. Remove unused Worker Groups and fleets. Consolidate temporary or duplicate Worker Groups. Increase the Leader's memory allocation to match the managed deployment size. Restart the Cribl Stream Leader service.

New known issue page(s) added: CRIBL-45296 | Outpost TLS distributed setting failing on validation due to presence of lakeAzure entry in the payload

New known issue page(s) added: CRIBL-45296 | Outpost TLS distributed setting failing on validation due to presence of lakeAzure entry in the payload

 I am currently stuck on the lab (CC Admin - Stream | Stream Projects and Security Lab), specially task 3.2 and 3.3. I was unable to add the team to the project with the Editor role because permissions field was grayed out and did not allow me to select or modify any options. 

SymptomAfter enabling Resiliency: Failover on a Cribl Stream Leader, the Leader restarts in Single mode and appears to have lost its configuration.The restart might complete without an obvious error in the Leader logs.When the configured failover path already contains configuration data, the Leader might instead start with unexpected or fresh configuration, an unexpected license, or without copying the current local configuration to the failover volume.EnvironmentCribl Stream self-managed distributed Leader Leader High Availability with Resiliency: Failover enabled Shared NFS failover volume located outside $CRIBL_HOMEResolutionStop the affected Cribl Stream Leader service before changing configuration files. Verify that the configured failover volume is mounted and accessible to the Cribl service account. Verify that the failover volume is outside the Cribl installation directory. If this is a new HA configuration, verify that the configured failover path is empty before enabling fail

SymptomWhen opening a Cribl.Cloud workspace page, the browser displays a 404 or "Workspace not found" page even though:The user can sign in to Cribl.Cloud. The workspace exists and is active. Requests in the browser Network tab return HTTP 200.The browser console may show errors such as:Cross-Origin-Embedder-Policy or Cross-Origin-Resource-Policy blocking errors. ERR_BLOCKED_BY_RESPONSE for requests to api.cribl.cloud.EnvironmentCribl.Cloud workspace access through a secure web gateway, proxy, or TLS-inspecting firewall. The proxy can modify response headers for Cribl.Cloud requests.ResolutionOpen the affected workspace page in a private/incognito browser window. Open the browser developer tools and select the Console tab. Reproduce the issue and look for COEP, CORP, or ERR_BLOCKED_BY_RESPONSE errors. In the Network tab, verify that the workspace-list request returns HTTP 200 and includes the expected workspace. Ask the network security administrator to bypass proxy inspection or respo

SymptomZscaler NSS Cloud reports latency delivering events through the Splunk HEC Source (in_splunk_hec) in Cribl Stream. Event batches accumulate before being acknowledged, even though Cribl Stream shows no resource saturation, backpressure, or rejected connections. Increasing worker group size does not resolve the latency.EnvironmentCribl Stream (validated 4.19.0; underlying mechanism is not version-specific) Source: Splunk HEC (in_splunk_hec), receiving from Zscaler NSS Cloud Sender: Any HEC client with a fixed concurrent-connection limit and serialized per-connection delivery (Zscaler NSS Cloud calls this setting UploadWorkers)ResolutionConfirm the mechanism using the spent value on completed log lines for the affected Source (channel: "input:in_splunk_hec"). A spent value that scales with event volume and correlates with the sender-reported latency confirms pipeline and route processing time is driving the delay. Reduce per-batch processing time in the Pipelines and Routes the eve

This release brings Apps to GA, puts AI throughout Cribl Search, adds tighter security controls across the control plane, and lets you search archived Splunk data without rehydrating it.Platform and AppsApp Backend Functions: Run searches, detect threats, and respond to events without the app staying open. AI Infrastructure Observability App: Track GPU usage and spend across environments, find idle capacity, and catch hardware issues early. Observability Apps Bundle: Ready-to-use monitoring for Kubernetes, uptime, databases, infrastructure, NetFlow, and real-user experience.StreamCribl Guard Privacy 3.0 model series: Better background detection, with Nano for speed, Balanced for the middle ground, and a new Pro option when accuracy matters most. Auto-parsing globally and in Routing: Auto-parse Sources globally, on selected Routes, or in the Parser function, with guardrails against duplicate processing. Auth Token Management for Node Connections: Rotate, audit, and revoke provisioning t

New known issue page(s) added: CRIBL-45304 | Microsoft Sentinel Destinations fail to authenticate with Microsoft

This guide walks you through pointing Claude Cowork’s OpenTelemetry (OTel) export to a Cribl Stream OTel Source in Cribl Cloud so you can monitor Cowork activity alongside your other telemetry. For full Cowork monitoring details, see Monitor Claude Cowork activity with OpenTelemetry and the Claude Cowork monitoring setup docs.Configure the Cribl OTel sourceIn Cribl Stream (Cribl Cloud): Create or open an OpenTelemetry source. Set: OTLP Version: 1.3.1 Protocol: HTTP Port: {PORT} (port 4318 is currently unavailable on Stream in Cribl Cloud; use a port in the 20000–20010 range) Turn on Extract logs and Extract metrics when you want OTLP batches split into per-event records for use in Stream pipelines. Enable TLS. Set Authorization > Auth Token to {TOKEN} (this will become the bearer token Claude Cowork uses). Note the port and token; you’ll plug these into Claude CoWork next. For more detail on source settings and ports, see the OpenTelemetry (OTel) Sourc

New known issue page(s) added: CRIBL-44852 | Load-balanced Sources fail to initialize after configuration changes in single-instance mode PLAT-15422 | Members with Read Only Permission on Cribl Insights can view Worker Group and Edge Fleet configurations Known issue page(s) updated: AI-4634 | Guard: \"Apply\" button adds all recommendations AI-4675 | Guard shows 'Samples expired' with visible samples and no way to regenerate recommendations AI-4808 | **MCP on Leader** toggle turns off when no LLM provider is configured CRIBL-25879 | When you teleport into a Windows Edge Node and navigate to Health, the CPU Load Average panel displays no results CRIBL-44054 | Failed SOCKS/TLS connection causes Worker/Edge Node to stay disconnected from the Leader CRIBL-44150 | Pack Syslog events go to DevNull after a Worker Process config reload CRIBL-44296 | File Monitor Sources in Packs ignore Pack Event Breaker Rulesets CRIBL-44454 | Enabled Cribl HTTP and Cribl TCP Source Auth Tokens are ignored on

Hi,I’m a certified Cribl Admin user and finished all the admin user labs and course.currently working on Cribl Stream certification, I’m stuck with an issue I cant complete "Stream | Stream Projects and Security Lab” as it mentions that” In order to enroll, you must complete the following: CC User Final Exam Section 3. “I have finalized all the CC user tutorials and exams and cant figure out how to solve this issue.Please I need your support

SymptomWhen you run a Cribl CLI command manually on Windows, it might not use the configuration stored in C:\ProgramData\Cribl. As a result, commands such as diag connect can default to localhost instead of the configured Leader or Outpost.For example, cribl.exe diag connect can return ECONNREFUSED for localhost:4200 even when the remote endpoint is reachable.PS C:\Program Files\Cribl\bin> .\cribl.exe diag connect Testing leader connectivity: localhost:4200 via TCP (plain) Config source: instance.yml FAILED: could not connect to localhost:4200 via TCP (plain) Error: connect ECONNREFUSED ::1:4200 connect ECONNREFUSED 127.0.0.1:4200The diag connect command resolves Leader connection settings in this order:Environment variables, including CRIBL_DIST_MASTER_URL instance.yml localhost, if neither provides the connection settingsEnvironmentCribl Edge on Windows Managed Edge deployment Windows MSI installation using the default paths: binaries in C:\Program Files\Cribl and runtime data, in

SymptomCribl Search returns a query_execution_error when querying Parquet files written by Cribl Stream from Office 365 Activity data. The error includes Code 117 and identifies the Message column:DB::Exception: Cannot parse JSON object here ... column: Message ... CriblParquetInputFormat ... (INCORRECT_DATA)EnvironmentCribl Stream writes Office 365 Activity events to an object-storage Destination in Parquet format. Cribl Search queries the resulting Parquet dataset. Office 365 events contain nested JSON values in fields that are also plain strings in other events. The Destination uses automatic or fixed Parquet schema handling.ResolutionNavigate to the Office 365 Activity Pipeline that processes events before the Parquet Destination. Add a Flatten Function to the Pipeline. Enter the top-level field containing nested objects, such as Message, in Fields. Configure the Depth and Delimiter for the expected nested structure. Save the Pipeline configuration. Identify every Route that sends

SymptomWhen you navigate to the Destinations page in Cribl.Cloud, the page fails to load and displays:Cannot read properties of undefined (reading 'endsWith')The page may load in an incognito or private browsing window.EnvironmentCribl.Cloud Destinations page Web browser with stale Cribl.Cloud cached site dataResolutionSign out of Cribl.Cloud. Clear the cookies and site data for cribl.cloud. Clear cached browser content, including cached JavaScript and CSS assets. Close open Cribl.Cloud tabs. Open a new browser window. Sign in to Cribl.Cloud. Navigate to the Destinations page.CauseThis can be caused by:Stale cached Cribl.Cloud JavaScript or CSS assets. Stale Cribl.Cloud cookies or site data.Additional InformationUse an incognito or private browsing window as a temporary workaround while clearing cached content and site data in the normal browser session.

ObjectiveIdentify whether delayed events originate in the downstream system, Cribl Stream Destination, Cribl Stream Source, or sending application.EnvironmentCribl Stream with an enabled streaming Source and Destination. Access to the downstream system and its ingestion or arrival metadata. Access to the sending application or forwarder host.ProcedureDefine the investigation window and identify a distinctive event. Record the event’s source, host, index or dataset, unique message content, event timestamp, and observed arrival time. Use the same identifying values at each investigation point. Check the downstream system. Search a broad time range around the expected arrival time. Verify whether the event exists with a delayed ingestion or arrival time. Review receiver health, ingestion rate, throttling, indexing, consumer lag, or equivalent downstream metrics. If the event exists downstream with a later ingestion time, investigate downstream processing or queueing. If the event is

ObjectiveConfigure a Cribl Stream remote Git repository without storing a plaintext password in .git/config.EnvironmentCribl Stream Leader On-premises deployment with an external Git repository Basic authentication or a credential-bearing Remote URLProcedureRotate the Git password or token if it may already have been exposed. On the Cribl Stream Leader, create an SSH key pair from the CLI as the Cribl service user. Use a passphrase-protected private key when required by your security policy. Add the public key to the Git provider with permission to access the repository. Configure the Cribl remote repository to use the provider’s SSH URL and SSH authentication. Do not include a username, password, or token in the Remote URL. Complete passphrase-protected SSH configuration through the Leader CLI, because this capability is CLI-only. Configure and verify SSH host-key handling for the Git provider, then test a Git operation such as commit and push. Confirm that the remote URL in .git/conf

SymptomIn Cribl Search, users added to a Usage Group cannot access a Dataset shared with a Team. Existing Team members can access the Dataset, while newly added users cannot.EnvironmentCribl Search in Cribl.Cloud Team-based Dataset permissionsResolutionVerify that each affected user already exists as a Member in the Organization. Navigate to Products > Organization > Members & Teams. Select the Teams tab. Open the Team that has access to the Dataset. Add each affected user under Team Members. Select Add. Select Save. Navigate to Cribl Search. Select Data > Datasets. Open the Dataset’s Share action. Verify that the Team has Read Only or Maintainer access. Verify that the affected users can view and search the Dataset.Note: Use Usage Groups to assign Search resource limits. Use Teams under Organization > Members & Teams to manage Team membership and Team-based access.CauseThis can be caused by:Adding users to a Search Usage Group instead of adding them to the Team tha

QuestionWhy does Cribl Stream File Monitor attempt to open unrelated subdirectory when Filename allowlist targets a file (e.g. /var/log) in the directory and Max depth is 0?EnvironmentCribl Stream or Edge File Monitor source Manual discovery mode on a Linux host Max depth configured as 0AnswerFile Monitor discovers filesystem entries under Search path before applying Filename allowlist.The allowlist filters files, not directories, so File Monitor can still call opendir on a subdirectory. If the Cribl service account lacks read permission, the source logs EACCES: permission denied.For a single target file, use its full path in both Search path and Filename allowlist. For example, use /var/log/messages in both fields.Additional InformationSee the File Monitor Source documentation for discovery, search-path, maximum-depth, and filename-allowlist behavior.