Skip to main content
Solved

S3 Source Not Returning Results Despite Matching Destination Path Configuration

  • April 8, 2026
  • 1 reply
  • 0 views
C
This message originated from Cribl Community Slack.
Click here to view the original link.

Hey, can anyone give me a pointer on what I may be missing in my S3 path. I have an s3 destination and am setting up a s3 source to "rehydrate" logs. I used the destination to populate the source as recommended and copied over the same partition expression I have in the destination but am not getting any results back. It is consistently being written to. Destination configurations - Bucket Name: archive path: palo_alto/${C.Time.strftime(Date.now()/1000, '%Y-%m-%d-%H:%M')} File name prefix expression: CriblOut File name suffix expression: .${__format}${__compression === "gzip" ? ".gz" : ""} Example from on premise s3 storage: /palo_alto/2026-04-07-00:01/CriblOut-zsf3kU.json.gz Source configurations: Bucket name: archive Path: palo_alto/${%Y-%m-%d-%H:%M}/${*} I've tried the path with and without the last subdirectory /${*} I've tried with relative time range and absolute time range that I've validated existed in storage.

Best answer by Stefan Laschitzki

If you still can, you might want to reconsider your partitioning expression. The current one will create a "flat" hierarchy will lots of folders under "palo_alto". It will be super expensive to replay from that bucket as each day up to 1440 new folders may be added to that folder. Listing them and deciding if they need to be considered will get costly soon. A "deeper" hierarchy will makes things much easier Amazon S3 Better Practices | Cribl Docs That said, I think this is caused by "too many quotes" in your Path expression and the missing "_time" prefix. Either put the hole expression into backticks or concatenate strings e.g. "palo_alto/${_time:%Y-%m-%d-%H:%M}/${filename}" You can find an example here Using S3 Storage and Replay | Cribl Docs and here S3 Collector | Cribl Docs

1 reply

S
If you still can, you might want to reconsider your partitioning expression. The current one will create a "flat" hierarchy will lots of folders under "palo_alto". It will be super expensive to replay from that bucket as each day up to 1440 new folders may be added to that folder. Listing them and deciding if they need to be considered will get costly soon. A "deeper" hierarchy will makes things much easier Amazon S3 Better Practices | Cribl Docs That said, I think this is caused by "too many quotes" in your Path expression and the missing "_time" prefix. Either put the hole expression into backticks or concatenate strings e.g. "palo_alto/${_time:%Y-%m-%d-%H:%M}/${filename}" You can find an example here Using S3 Storage and Replay | Cribl Docs and here S3 Collector | Cribl Docs
Terms & ConditionsAccessibility statement