High CPU and Event Loop Utilization on Cribl Edge Caused by File Monitor Source Scanning Overhead

Symptom

Cribl Edge worker process (w0) shows sustained high CPU usage (above 100%) and elevated event loop utilization (ELU), above 50%, but actual data throughput is very low (for example, less than 5 MB/minute). A CPU profile confirms that the majority of CPU time is consumed by discoverManual() filesystem scanning operations and garbage collection rather than data processing.

Environment

• Cribl Edge
• File Monitor sources

Resolution

1. Identify which File Monitor sources are actively collecting files and which are not. Check the source metrics in the Edge UI or diagnostic bundle. Sources showing zero tracked files are consuming CPU on directory scanning without producing data.
2. Disable any File Monitor sources that are not collecting files on the affected nodes. Each disabled source eliminates an entire scan cycle and its associated garbage collection overhead. This is typically the highest-impact change.
3. Increase the polling interval on remaining File Monitor sources. The default is 10 seconds. Increase to at least 30–60 seconds for most sources. For sources that do not require near-real-time collection can be pushed higher (e.g., 300 seconds). This reduces scanning frequency.
4. Replace recursive double-star (**) glob patterns in the Filename Allowlist with explicit depth-level patterns where the directory structure is known. For example, replace **/errors.log with */errors.log or */subdir/errors.log. Recursive globs force a full directory tree scan on every scan cycle.
5. Verify the Search Path field on each File Monitor source contains only a real directory path with no wildcards. The Search Path field does not support wildcards. A path such as /apps/logs/* causes an ENOENT error because lstat treats the asterisk as a literal directory name. Wildcards belong only in the Filename Allowlist. See the File Monitor Source documentation for reference.
6. Where multiple File Monitor sources scan the same or overlapping directory trees, consider consolidating them into fewer sources with a broader Filename Allowlist. Each source triggers separate scanning overheard. Reducing the number of sources reduces the total number of directory traversals per scan cycle.
7. Set the depth (Max Depth) configuration on File Monitor sources to limit how deep directory traversal goes when the exact directory structure is known.

Cause

Cribl Edge runs a single worker process (w0). Each enabled File Monitor source in manual discovery mode runs discoverManual() on every polling interval to walk the configured search path, evaluate filename allowlist patterns, stat files, and check log state. When many sources are enabled with aggressive polling intervals (for example, 10 seconds), broad search paths, and recursive glob patterns, the cumulative scanning overhead saturates w0's CPU and ELU.

The overhead has three main components:

• Filesystem scanning: discoverManual() is used for every directory and file in the search path on every polling cycle.
• Garbage collection: Each scan cycle creates large numbers of short-lived objects. These accumulate and trigger frequent garbage collection passes. In a CPU profile, GC can consume 25-30% of total CPU time.
• Redundant scanning: Sources that find zero files still perform the full directory traversal on every cycle. Misconfigured search paths with wildcards generate ENOENT errors on every scan or hourly discovery cycle without ever collecting data.

Last Validated

4.16.1

Additional Information

For more information, see File Monitor Source documentation.