Skip to main content
Question

How do I extract AWS EC2 metadata from an Edge File Collector's metadata fields?

  • March 11, 2025
  • 1 reply
  • 27 views
D

I am running Edge Nodes on our EC2 instances.

When reviewing internal fields from our File Collector sources, I noticed that there is alot of really useful data that our SREs use to investigate which regions & servers are experiencing issues during incidents.

These include:

- AMI image

- Region

- Availability Zone

- Architecture

- Public IP

- Private IP

I'd like to extract these internal fields as event fields that get sent to our destinations. What is the simplest way to do that?

    1 reply

    Jeremiah Ross

    Hi Dean,

    It sounds like an pre-processing pipeline that includes an eval function will do the trick. I took a few screenshots from the lab to help you along.

    1. Review Your metadata. For an example, see the below fields nested in __metadata.aws
    765_19c5e3513a904a9e9e494f86fbc612f6.png

    2. Create a new pipeline that can be assigned as the source's pre-processing pipeline

    3. Create an eval function in your new pipeline

    4. Configure your eval function like this:

    Filter: true (to catch all events) - or -

    Filter: source.includes('file') ( to filter by file)

    Eval:

    • Name: <desired field name>
    • Value Expression: __metadata.aws.<desired field to extract>
    765_c9dea14beec3407b93a46f186cb40d5c.png

    5. Add this new pipeline to your source (after testing)

    765_f07bab669a18453ebd1b23e66e46733b.png

    6. Commit & Deploy to save changes

    I hope that this helps

    Terms & ConditionsAccessibility statement