Ingest Orphaned Staging Files in Cribl Stream

Objective

Recover orphaned files left in a staging directory after a Worker Process (WP) crash (e.g., OOM) by rebuilding the state file so Stream re-processes them on the next startup.

Environment

• Cribl Stream 4.x
• File-based output destinations using staging directories

Procedure

1. Identify the orphaned files

Locate the staging directory for the affected output and confirm orphaned .tmp files exist:

STAGE_DIR="$CRIBL_HOME/state/outputs/staging/<output_id>" find "$STAGE_DIR" -name "*.tmp" -o -name "*.tmp.agg.*"

Note the current worker process ID from the existing state file name (e.g., CriblOpenFiles.0.json → worker ID is 0).

2. Stop Stream

$CRIBL_HOME/bin/cribl stop

3. Rebuild the state file

Back up the existing state file for safekeeping:

cd $STAGE_DIR mv CriblOpenFiles.0.json CriblOpenFiles.0.json.bak

Review the backup so you know what the normal structure looks like. The format is:

{ "files": [ { "file": "/opt/cribl/state/outputs/staging/<output_id>/path/to/file.tmp", "state": "closing", "retryNum": 0 } ] }

Create a new state file that lists of the staging directory files and sets the state set to "closing". You can build it manually or use this script:

#!/bin/bash WORKER_ID="0" # worker process ID from your active state file name STAGE_DIR="/opt/cribl/state/outputs/staging/<output_id>" STATE_FILE="${STAGE_DIR}/CriblOpenFiles.${WORKER_ID}.json" echo '{"files":[' > "$STATE_FILE" first=true find "$STAGE_DIR" \( -name "*.tmp" -o -name "*.tmp.agg.*" \) | while read f; do if [ "$first" = true ]; then first=false; else echo ',' >> "$STATE_FILE"; fi echo "{\"file\":\"$f\",\"state\":\"closing\",\"retryNum\":0}" >> "$STATE_FILE" done echo ']}' >> "$STATE_FILE"

Replace <output_id> with the actual output ID (the subdirectory name under staging/).

Setting state: "closing" tells the WP these files are ready to upload. On startup, it transitions them to dead-closing and begins processing after a 30-second delay.

4. Start Stream

$CRIBL_HOME/bin/cribl start

5. Validate

Confirm the files were sent:

• The staging directory — orphaned files should be removed after successful upload.

• Logs for upload activity:

  tail -f $CRIBL_HOME/log/cribl.log | grep -E "(moveToFinal|upload|ingest|dead-)"

• The state file — a successful run will remove processed entries from the files array.

• Files that exceed 20 retries will be moved to $STAGE_DIR/dead-letter/ — investigate these separately.

Additional Information

• State file worker ID: The worker ID in the state file name must match the running WP's ID. Check $CRIBL_HOME/log/cribl.log if unsure.

• Stale aggregation claims: If files in the .aggregate/ subdirectory have .agg.{N} suffixes from a dead worker, strip the suffix before rebuilding the state file:

  find "$STAGE_DIR/.aggregate" -name "*.agg.*" | while read f; do mv "$f" "$(echo "$f" | sed 's/\.agg\.[0-9]*$//')" done

• Same-node recovery: This procedure works on the same healthy worker node. No file copies or additional nodes are needed if the instance is not being replaced.

• Instance at risk of replacement: If the node may be replaced before recovery, copy the staged files to persistent storage (NFS) first and perform the recovery on a stable node.

• Why files become orphaned: Stream discovers staged files exclusively through the state file — not by scanning the directory. When a WP crashes before flushing state, the replacement process has no record of the in-flight files.

Updated Resolution - 4.17.1 and Later

Since 4.17.1, file-based destinations ship with an orphan file reconciler that does this recovery automatically. It reads every CriblOpenFiles.*.json in the staging dir, walks the staging tree, and anything that isn't tracked by any worker and is older than 5 minutes gets queued for close/upload through the normal retry and dead-letter path.

It's off by default on destinations that existed before the upgrade (new ones get it on). So for a setup on 4.17.1+, the steps would be:

1. Open the destination, go to Advanced Settings, and turn on "Enable orphan file recovery". You can leave "Period (minutes)" at 10 unless they want it faster (minimum is 5). For Cribl Lake destinations, there is no UI toggle. Instead, edit the destination config with “Manage as JSON” and update/add the following:
   ], "orphans": { "disabled": false, <-- false enables orphan recovery "periodMin": 10 <-- scan for orphans every 10 minutes } }
2. Commit and deploy.
3. Wait one period. The first scan runs shortly after the config lands, then every N minutes. Files younger than 5 minutes are skipped on purpose so we don't grab something a worker is still writing; if "Remove empty staging directories" is on, the current, previous and next time buckets are skipped for the same reason.
4. Validate:
   • cribl.log shows "Creating orphan file reconciler" for the destination on each worker that runs it (with "Remove empty staging directories" on, every worker scans its share of buckets; otherwise only worker 0 scans the whole dir).
   • The destination's stats get an "orphans" block; "processed" is how many orphans were queued in that interval.
   • The staging dir drains as uploads succeed. Anything that keeps failing ends up in dead-letter/ after the usual retries, same as today.