S3 is a strong destination for data that still matters but does not typically need to be searched in real time. Compliance retention, backup and replay, data lake workflows, cost-conscious storage for high-volume data you are not ready to throw away. S3 handles all of it.
With Cribl in front of it, S3 gets a lot more useful. Cribl filters, shapes, partitions, and routes data before it lands, so what ends up in the bucket is actually structured and consumable, not just a pile of compressed noise.
Two destination options in Cribl Stream
Amazon S3 Compatible Stores is the general-purpose choice. Use this for standard logging and telemetry workflows.
Data Lake Amazon S3 is for teams who need finer control over output format and schema.
What you need before you start
- An S3 bucket and its region
- IAM permissions on that bucket (full permission list in the docs below)
- Authentication: access key and secret, instance profile, or an IAM role for Assume Role
Assume Role is the better pattern for most production environments. If you are on Cribl.Cloud or using Hybrid Workers, see the Securely Connecting blog post below for the exact setup.
What to route to S3
Good candidates:
- Compliance logs you need to retain but rarely search
- High-volume data for trend analysis that does not need real-time indexing
- Backup copies of data going to a more expensive destination
Before you route anything, decide on your path and partitioning structure. This is harder to fix later. The Better Practices doc below covers partitioning, cardinality, and file management in detail.
Troubleshooting
Most S3 destination problems fall into four buckets:
- Events are not matching the route filter
- IAM permissions are wrong or incomplete
- Bucket name, region, or path expression has an error
- Batching, format, or compression settings do not match what downstream tools expect
Check the destination status and internal logs in Cribl first. Then check AWS permissions. The common errors article below covers the usual suspects.
Resources
Knowledge Base
- Article - Common AWS S3 Errors (Source and Destination)
Cribl Docs
- Amazon S3 Compatible Stores Destination -- setup, permissions, encryption, and testing
- Data Lake Amazon S3 Destination -- auth, Assume Role, output schema, and advanced options
- Amazon S3 Better Practices -- partitioning, cardinality, open file management, and tuning
Cribl Blog
- Setting Up and Tuning Amazon S3 as a Cribl Stream Destination -- setup fields, staging behavior, partitioning, and scaling
- Securely Connecting an Amazon S3 Destination to Cribl.Cloud and Hybrid Workers -- Auto auth, Assume Role, and the AWS-side role and policy setup
-
Thou Shall Pass! Troubleshooting Common Amazon S3 Errors in Cribl Stream - In-depth article on troubleshooting issues with S3 destinations.