Cribl Stream Amazon S3 SQS Source Shows No Data

Symptom

An Amazon S3 Source (SQS-based) in Cribl Stream shows as healthy, but no events arrive at the expected destination for newly created Snowflake log files. The customer observes:

• The S3 bucket contains Snowflake log objects, but the Cribl route or destination receives no data.
• The SQS queue configured for the S3 Source shows no messages for the expected time window.
• After re-uploading a test file to the S3 bucket, data begins to flow, confirming that ingestion works only when a new object lands in S3.
• In some cases, captured events are visible in Live Data but are routed to devnull instead of a real destination.

Environment

• Product: Cribl Stream
• Deployment type: Managed AWS
• Source: Amazon S3 Source using SQS queue notifications

Resolution

1. Confirm that the S3 bucket is configured to send object-created notifications to SQS

   1. In the AWS console, open the S3 bucket used for Snowflake logs.

   2. Go to Properties > Event notifications.

   3. Verify there is an event notification that sends s3:ObjectCreated:* events for the relevant prefix (folder) to the SQS queue used by the Cribl Amazon S3 Source.

2. Verify that the SQS queue receives messages for new S3 objects

   1. In the AWS console, open the SQS queue configured in the Cribl Amazon S3 Source.

   2. Check the Number of messages available metric.

   3. If the queue has 0 messages and only historical objects exist in the bucket, note that S3+SQS notifications are generated only for new object creations, not for files that were already present before notifications were enabled.

3. Trigger a test upload to populate the queue and confirm ingestion

   1. Upload or re-upload a small test file into the S3 bucket under the path that is covered by the event notification.

   2. Refresh the SQS queue metrics and confirm that at least one new message appears.

   3. In Cribl Stream, open the Amazon S3 Source and use the Logs and Status tabs to confirm that the source:

      1. Successfully connects to the SQS queue.

      2. Receives notifications and pulls objects from S3.

   4. Use Live Data on the worker group to confirm that events from the test file are now visible in Cribl.

4. Configure a route so data does not go to devnull.

5. Re-test end-to-end delivery

   1. Upload another log file into the S3 bucket.

   2. Confirm that:

      1. The SQS queue receives a new message for the object.

      2. The Cribl Amazon S3 Source processes the notification and pulls the object.

      3. The configured route sends events to the expected destination (for example, CrowdStrike or an intermediate destination used for the Snowflake-to-CrowdStrike flow).

Cause

This issue may be caused by he S3 bucket not generating any s3:ObjectCreated:* notifications for the configured SQS queue, so the queue is then empty and the S3 Source has no messages to process.

Additional Information

Enable debug logging on the source by setting the log level to debug in the source configuration. Messages to look for:

1. Verify the SQS queue is receiving notifications by checking for these debug log messages:

   • "received SQS messages" — confirms the poller is receiving messages from the queue.

   • "received object notification" — confirms the notification was parsed and a bucket/key was extracted.

   • "download and processing" — confirms Cribl is attempting to fetch the object from S3.

   • "Processing complete" — confirms the file was successfully downloaded and processed.

2. If none of the above messages appear, verify that the SQS queue has messages by checking the AWS SQS console or re-uploading a file to the S3 bucket to trigger a new notification.

3. If "received object notification" appears but "Processing complete" does not, check for error messages.