Is there any info on Search costs when searching AWS S3 buckets? Specifically on how data transfer works. For example, does Search hit S3 from US East and is $.01 per GB against my AWS account? Is Search caching at all? https://aws.amazon.com/s3/pricing/?p=pm&c=s3&z=4
Original post was from https://cribl-community.slack.com/archives/C03HK1K33FA/p1678133798883439
Question
Is there any info on Search costs when searching AWS S3 buckets?
On the AWS side, you pay for API calls to the bucket. Assuming the bucket is in the same region as the Cribl.Cloud instance, there is no data transfer fee. Assuming you're using S3 standard storage tier, the API calls we would be making:
LIST: $0.005/1000 requests
GET: $0.0004/1000 requests
My region via the overview page says us-west-2. Does this mean it is to my advantage to host my s3 in us-west-2?
Okay, thanks. Also, are there any logs showing the buckets and/or objects being hit from Search? I only saw logs from stream, edge, etc.
S3 access logs should be visible in the Search Status page, and when searching the internal Search logs themselves
+1Use case you mention above, if I am not searching, will I get charged api hits? consistantly?
you only get charged API hits when you actively run a search. There's no background or periodic updates that occur at this time.
+1This definitely helps with understanding how it works, thanks!
we've actually added to the docs some information for clarity here: https://docs.cribl.io/search/set-up-s3#data-transfer-charges
Sign up
Already have an account? Login
Login to the community
No account yet? Create an account
Using your Cribl Curious or University Account
User Login Employee loginEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.