Populating lookup table using incoming events

You can use the Redis function to update Redis-based data from within a pipeline using live data.

This isnt currently possible with CSV-based lookups.

Weve created a product with custom functions that will read/write tables in data stores, which can be mongo, Oracle, MySql, MSSQL, Postgress, DB2 and others, making the data available across workers. Ping me if this aligns with what youre looking for.

You can optionally configure a script in "Sources" to run and collect the data you need from the source and output the results to /opt/cribl/data/lookups. This is handy in the instance you have a fileserver hosting reference data or an API endpoint that has the file or data ready to be consumed.
Example script for hitting endpoint and writing to lookup directory.

wget URL-to-Data -O /opt/cribl/data/lookups/filename.csv

I was able to do this with a lookup table I uploaded then pulling the information from a source system and using the following api calls:
staging the lookup update: /api/v1/m/default/system/lookups?filename=filename.csv
in the body only passing the string value of what I want in the new filename
Part of the output will have a temp file name use that to pass into this call:

updating the file: /api/v1/m/default/system/lookups/s1_ip_lookup.csv
{
  "id": "filename.csv",
  "fileInfo": {
    "filename": ""filename.csv.x1JAmRt.tmp""
  }
}

Commit and Deploy!