Is there a limit to file size when importing data from object stores?

Individual event size limitations are controlled by the event breaker definitions applied to the Source. Default is 50 KB max size. Overall files (objects) are not limited per se, but remember that objects have to be downloaded before events can be extracted. And while decompression is done in a streaming fashion so disk space wont be all that impacted, a large file can take more CPU cycles than expected.

Max event size is predicated on the breaker used. The default breaker supports event size up to 51,200 bytes. If events are expected to be bigger than that the Cribl - Do not break ruleset can be used, which supports up to 1MB events, but if that doesnt properly break the events being replayed then a completely custom breaker will be required. Max event size for a customer breaker is 134,217,728 bytes.

As for max file size, no limit exists based on the info from the Software engineering team.

One thing that Ive noticed is that if you have a very large raw payload from an S3 collector (say like an event that is 20+MB), there seems to be some issue with the API worker process on the Leader being able to handle that.

Yeah Im aware of that issue that Jeremy is working on with you. Were still investigating that so well have more information once we determine the root cause of that. But at present there arent any Cribl defined limits on the files but there is for the event size.