Skip to main content
Question

Query on simplifying JSON to CSV conversion.

  • Mar 11, 2025 3:13 AM UTC
  • 12 replies
  • 128 views

If I am reading the docs correctly, when converting from json to CSV, I have to define all the fields. Since all of the fields are really already known, is there a way to output all the fields first without having to define all of them? We are working on converting some Azure logs, and there are LOTs of properties.field(s). Hoping there is another way without us having to define all of these in the "Fields to serialize" field within the Serialize function.

12 replies

Jon Rust
Forum|alt.badge.img
  • Employee
  • Mar 11, 2025 4:44 AM UTC

All fields must be listed, in the order you require.Copy-pasta works for the list. Possibly in your analysis tool of choice you can get a list?


  • Author
  • Employee
  • Mar 11, 2025 4:44 AM UTC

ok, tahnks


  • Author
  • Employee
  • Mar 11, 2025 4:44 AM UTC

thanks


Jon Rust
Forum|alt.badge.img
  • Employee
  • Mar 11, 2025 4:44 AM UTC

create a temp function to serialize into K=Vand another to use regex to pull all the field names. In the preview window, copy-pasta the list


Jon Rust
Forum|alt.badge.img
  • Employee
  • Mar 11, 2025 4:44 AM UTC

hold on, trying it out


Jon Rust
Forum|alt.badge.img
  • Employee
  • Mar 11, 2025 4:44 AM UTC

```{ "conf": { "output": "default", "streamtags": [], "groups": {}, "asyncFuncTimeout": 1000, "functions": [ { "filter": "true", "conf": { "mode": "extract", "type": "json", "srcField": "raw", "dstField": "_raw" }, "id": "serde" }, { "filter": "true", "conf": { "type": "kvp", "fields": [ "!", "!cribl_breaker", "" ], "dstField": "raw", "cleanFields": false, "srcField": "_raw" }, "id": "serialize" }, { "filter": "true", "conf": { "source": "_raw", "iterations": 100, "overwrite": false, "regex": "/(?<fields>[-a-zA-Z0-9]+)=/g" }, "id": "regex_extract" }, { "filter": "true", "conf": { "add": [ { "disabled": false, "name": "fields", "value": "fields.join(',')" } ] }, "id": "eval" } ] }, "id": "list_fields"}```


Jon Rust
Forum|alt.badge.img
  • Employee
  • Mar 11, 2025 4:44 AM UTC

» my sample data started as stringified JSON» i parsed it into actual JSON» then serialized into K=V» then regex extract for all field names (strings leading up to `=`), with global flag set» then eval to join the fields array into a list


Jon Rust
Forum|alt.badge.img
  • Employee
  • Mar 11, 2025 4:44 AM UTC

obviously nested json would present a problem, but then i don't know how you'd plan on making that CSV in the first place


  • Author
  • Employee
  • Mar 11, 2025 4:44 AM UTC

great. Let me give that a try


  • Author
  • Employee
  • Mar 11, 2025 4:44 AM UTC

the nested is going to be the issue


Jon Rust
Forum|alt.badge.img
  • Employee
  • Mar 11, 2025 4:44 AM UTC

how would you do nested in CSV?


  • Author
  • Employee
  • Mar 11, 2025 4:44 AM UTC

not sure. I think we are doing this for another source, so I need to look. We will tackle this next week