Skip to main content

I am running Edge Nodes on our EC2 instances.

When reviewing internal fields from our File Collector sources, I noticed that there is alot of really useful data that our SREs use to investigate which regions & servers are experiencing issues during incidents.

These include:

- AMI image

- Region

- Availability Zone

- Architecture

- Public IP

- Private IP

I'd like to extract these internal fields as event fields that get sent to our destinations. What is the simplest way to do that?

Hi Dean,

It sounds like an pre-processing pipeline that includes an eval function will do the trick. I took a few screenshots from the lab to help you along.

  1. Review Your metadata. For an example, see the below fields nested in __metadata.aws
765_19c5e3513a904a9e9e494f86fbc612f6.png

2. Create a new pipeline that can be assigned as the source's pre-processing pipeline

3. Create an eval function in your new pipeline

4. Configure your eval function like this:

Filter: true (to catch all events) - or -

Filter: source.includes('file') ( to filter by file)

Eval:

  • Name: <desired field name>
  • Value Expression: __metadata.aws.<desired field to extract>
765_c9dea14beec3407b93a46f186cb40d5c.png

5. Add this new pipeline to your source (after testing)

765_f07bab669a18453ebd1b23e66e46733b.png

6. Commit & Deploy to save changes

I hope that this helps

Reply


Terms & ConditionsCookie settings