Cribl Search Federated v1 Queries Fail When summarize Has Very High Cardinality

Symptom

A Search query that uses summarize ... by across too many unique values can fail on federated v1 datasets because the coordinator must hold a very large aggregation state in memory during merge, which can lead to a V8 heap out-of-memory failure. A scheduled or ad-hoc search fails even when the query syntax is valid.

Environment

• Cribl Search
• Federated v1 datasets
• Large or high-cardinality aggregation queries

Resolution

1. Reduce the group-by cardinality by removing or rethinking the highest-cardinality fields in the summarize ... by clause.
2. Add pre-filters or shorten the time range so fewer unique groups must be merged.
3. If the dataset is federated v1, move the workload to v2 and rerun the query.
4. Verify the rerun completes successfully after those changes.

Cause

In federated v1 Search, very large group-by result sets can force the coordinator to keep too many aggregation buckets in memory during the final merge stage. In the issue reviewed here, grouping by srcaddr and query_name over roughly 47 GB / 88 million VPC DNS events caused that memory pressure and failed the job.

Additional Information

In the sample issue, moving the dataset from v1 to v2 allowed the same query to complete successfully in 216 seconds.