Reduce Cribl Edge File Monitor CPU Usage with Discovery Optimizations

Symptom

Cribl Edge uses high CPU when a File Monitor source repeatedly scans a broad filesystem path for a specific log file. The source may walk directories on every polling interval even when most directories contain no matching files.

Environment

• Cribl Edge
• File Monitor source using Manual discovery mode
• Cribl Edge 4.20.0 or later when using Enable discovery Throttling

Resolution

1. Navigate to Products > Edge.
2. Under Fleets, select the target Fleet.
3. Select More > Sources.
4. Select File Manager.
5. Set Discovery Mode to Manual.
6. Set Search path as close as possible to the directory containing the target log file.
7. Set Max depth to the smallest value that includes the target log file.
8. Increase Polling interval when near-real-time discovery is not required.
9. Configure Discovery Optimizations according to the directory layout:
   • Enable discovery Throttling for broad searches that still require directory traversal. Start with the default target of 25%.
   • Enable Optimize for leaf directories when matching files are distributed across folders that stop changing over time.
10. Verify that the target file is discovered within an acceptable time.

The Filename allowlist filters file paths after discovery. It does not prevent File Monitor from descending into directories. Only Search path and Max depth limit the scope of the directory walk.

Both discovery-optimization settings apply only when the File Monitor source uses Manual discovery mode.

Optimize for leaf directories

• File Monitor uses directory modification time and exponential backoff to reduce rescans of unchanged directories that previously contained matching files.
• The backoff can reach 64 polling cycles between full rescans.
• Directories without matching files, or directories containing only subfolders, continue to be scanned on every pass.
• A directory modification time changes when files are created, deleted, or renamed. Appending data to an existing file does not change the directory modification time.
• This option can delay detection of new data.

Enable discovery Throttling

• Discovery runs in 200 ms cycles, split into a work slice and a pause.
• With the default target of 25%, Edge processes directory entries for 50 ms and then yields for 150 ms.
• The target can be set from 1% to 99%.
• Lower values reduce CPU spikes but increase discovery time.
• Throttling changes how quickly discovery runs, not which files it finds.
• Work performed on files that have already been matched is not included in the cap.

For a recursive search for one specific file, Optimize for leaf directories provides little benefit when most scanned directories do not contain matching files. Enable discovery Throttling reduces CPU spikes, but it spreads the same discovery work over a longer period rather than removing it.

File Monitor does not remember a discovered file path and stop searching. In Manual mode, each Polling interval starts discovery from the Search path again.

Cause

This can be caused by:

• A broad Search path, unlimited Max depth, and short Polling interval causing repeated large directory walks.
• Recursive filename patterns causing full directory-tree scans on every discovery cycle.
• File Monitor sources scanning the same or overlapping directory trees.
• File Monitor sources that find no files but still perform the full directory traversal on every cycle.
• Recursive searches in which most directories do not contain matching files. These directories are scanned on every pass and are not reduced by Optimize for leaf directories.

Additional Information

• File Monitor Source documentation
• Cribl Stream 4.20.0 release notes