This message originated from Cribl Community Slack.
Click here to view the original link.
As we had some debate about this today in a Cribl class... how does Search run its workloads?
was it Lambda functions in the early times, and nowadays it's... a Stream instance standing on standby for this? Is this a shared instance between different orgs, or a dedicated one? I've read the part about parallel workers and saw their logs (and saw that they log as version 4.18.1, which suggests at least shared codebase with "general" Cribl) - but are there more details? I'm curious :smile:
Solved
Cribl Search: Differences between v1 and v2 workloads execution
Best answer by David Cavuto
Hi @user. At a high level, I can characterize the difference in v1 and v2 Search by comparing the two functionally.
v1 Datasets execute with massively distributed, dynamic set of compute Executors. They spin up when a Search is executed, run for the duration of the search (or until they hit their timeout), and then die. New ones spin up as necessary. This process continues until the search is complete. Each Executor process is responsible for exactly one object store file at a time, under most conditions. Data streams back to the Search Coordinator from every executor via secure sockets, and are additionally processed there, then sent to the user. Once the search is done or cancelled, all executors are killed or die on their own.
v2 Datasets execute with a monolithic code execution environment. It runs in a multithreaded, multicore platform called the "Federated Engine" that spins up once, and then remains active for several hours, depending on usage. The v2 code on the Federated Engine is a different codebase than the v1 Executors, optimized for the Federated Engine platform. Many object store files are processed in parallel. Similarly to v1, data streams back to the Search Coordinator, additionally processed, and then sent to the user. The v2 compute platform stays active for many minutes up to many hours, dynamically determined based on Search load (the more Searches that have been executed, the longer the compute stays awake). Once the Search load goes to zero and the timeout is hit, the Federated Engine goes to sleep and remains there until the next search is executed. It takes about 20 seconds to wake up the next time a Search is executed. In tests, the v2 execution environment is considerably faster (3x-5x) than the maximum configuration of v1 Executors, when running the same search on the same object store data, and in some cases is even faster than that.
The code versions reported in the log are intentionally synchronized across all execution environments, regardless of platform or codebase, so you can't really infer anything from that.
I hope that at least partially gives you some insight.
Not sure what information we can share :blob_smile:
There's a reference to what's being done for v2 here: https://cribl.io/blog/new-engine-under-the-hood-cribl-search-v2s-faster-federated-architecture/
For the question you raise about management, the "Each organization contains a dedicated environment within Cribl.Cloud" logic applies.
If you'd like to speak with the Search team, I'd say put it on your CriblCon agenda for sure :sparkling_heart: There's a lot of cool stuff being done and whilst we don't share all of it publicly, you'll be able to get more insight into what's happening.
ha, I wish I could make it to Con... but if you happen to run some event on the other side of the ocean, that could actually work :slightly_smiling_face:
Hi @user. At a high level, I can characterize the difference in v1 and v2 Search by comparing the two functionally.
v1 Datasets execute with massively distributed, dynamic set of compute Executors. They spin up when a Search is executed, run for the duration of the search (or until they hit their timeout), and then die. New ones spin up as necessary. This process continues until the search is complete. Each Executor process is responsible for exactly one object store file at a time, under most conditions. Data streams back to the Search Coordinator from every executor via secure sockets, and are additionally processed there, then sent to the user. Once the search is done or cancelled, all executors are killed or die on their own.
v2 Datasets execute with a monolithic code execution environment. It runs in a multithreaded, multicore platform called the "Federated Engine" that spins up once, and then remains active for several hours, depending on usage. The v2 code on the Federated Engine is a different codebase than the v1 Executors, optimized for the Federated Engine platform. Many object store files are processed in parallel. Similarly to v1, data streams back to the Search Coordinator, additionally processed, and then sent to the user. The v2 compute platform stays active for many minutes up to many hours, dynamically determined based on Search load (the more Searches that have been executed, the longer the compute stays awake). Once the Search load goes to zero and the timeout is hit, the Federated Engine goes to sleep and remains there until the next search is executed. It takes about 20 seconds to wake up the next time a Search is executed. In tests, the v2 execution environment is considerably faster (3x-5x) than the maximum configuration of v1 Executors, when running the same search on the same object store data, and in some cases is even faster than that.
The code versions reported in the log are intentionally synchronized across all execution environments, regardless of platform or codebase, so you can't really infer anything from that.
I hope that at least partially gives you some insight.
thanks for sharing, yes, that gives me an idea :slightly_smiling_face: I've only toyed aronud with Search a little, but I love to know about the nitty-gritty inner details :slightly_smiling_face:
are there any plans to make Search available on-prem, at any point?
@user we've been asked for that feature a lot. What would your requirements for that feature look like?
It was more of a generic questions... I'm doing consultancy/PS work in :de: , and most of our Cribl customers are still on-prem. A bunch of them likely always will be. So that's why "will this be available on-prem" is usually my first question because often it is often their first question :slightly_smiling_face:
most of those customers use Cribl together with Splunk (and Elastic is becoming more relevant) :slightly_smiling_face:
That tracks. In future, if you can gather any feedback I'm sure David would love to know more. Customer feedback is great :star-struck:.
For search specifically understanding what they'd like to use search for (Which datasets) and what their key problems to solve actually are would be ideal.
If you tell me that this is generally an option, and you're not saying "Search will always be Cloud-only" - sure, I'll discuss this with customers when it comes up :slightly_smiling_face:
And also what storage mechanisms they'd like to use? (on-prem S3?) Also, is a cloud leader desirable or permissible or does it have to be entirely on-prem?
Sign up
Already have an account? Login
Login to the community
No account yet? Create an account
Using your Cribl Curious or University Account
User Login Employee loginEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.