Skip to main content
Solved

Logstream Filesystem output

  • March 11, 2025
  • 2 replies
  • 17 views
S

The Filesystem destination allows setting a prefix and suffix for the filename. However, I don’t see a place to set the actual base filename, or to just not have one. I don’t want the 6 random character base filename. How can this be removed? Also, I’d prefer not to have file staging, since I am not compressing files, is there a way to just write raw output to disk in real-time? Using the staging dir creates latency.

Best answer by Harry Gardner

Randomness is required in filenames destination to avoid race conditions as there are multiple worker processes writing data to files. As Dritan mentions, it is also required to avoid conflicts when moving to the final directory. The filesystem output is optimized for speed the extra logic required to move vs. append to the output file would have a negative impact on performance.

The staging directory is required because the filesystem destination is required to buffer many megabytes of data per file and worker process before moving to the final destination. Think of the staging directory as a filesystem backed buffer. If we remove the staging directory this destinations memory footprint would be greatly increased as the events would need to be buffered in memory.

    2 replies

    D
    • dritan
    • Employee
    • March 11, 2025

    IIRC the random 6 chars in the middle of the filename are needed by the application to ensure moving to the final directory happens without conflicts.
    Staging directories are required for non-streaming destinations like Filesystem, S3 etc. Whats the requirement for latency in this case? You can tweak Advanced Settings or or if the consuming application can receive events in real-time I suggest you consider another type of Destination such as TCPJSON etc.

    H
    • Harry Gardner
    • Employee
    • Answer
    • March 11, 2025

    Randomness is required in filenames destination to avoid race conditions as there are multiple worker processes writing data to files. As Dritan mentions, it is also required to avoid conflicts when moving to the final directory. The filesystem output is optimized for speed the extra logic required to move vs. append to the output file would have a negative impact on performance.

    The staging directory is required because the filesystem destination is required to buffer many megabytes of data per file and worker process before moving to the final destination. Think of the staging directory as a filesystem backed buffer. If we remove the staging directory this destinations memory footprint would be greatly increased as the events would need to be buffered in memory.

    Terms & ConditionsAccessibility statement