Data Health Monitoring in Cribl 📈
Monitoring how data flows through a Cribl deployment is essential to maximizing the value of your data. This includes considerations like traceability of any changes alongside assuring data integrity. This article covers a few considerations for how to go about achieving your validation and continuous assurance objectives.
Before you start - Understand your data flow objectives
Before you do any complicated analysis, your primary initial goal should be to understand your Sources and Destinations. This is particularly important if you are interacting with a Cribl deployment which is unfamiliar to you. Can you understand the system intent and objectives? Data will often be routed, transformed, duplicated or enriched. By understanding what the engineers and architects were attempting to do you'll be able to look more closely at the logic they used to achieve this.
At this point Cribl Insights is great for getting a high level view of data flows, sizing and directionality. You should be able to see the core branching logic and any notable changes to event numbers which could indicate sampling, aggregation or conditional drops. Changes in volume without changes in event count may indicate schema changes or enrichment activities.
Methodical testing is the best - Capture some data!
Once you’ve established the structure of your data flows, you can then start systematically capturing samples of data and testing each pipeline. Having up to date defined captures for each source is highly valuable. These samples support pipeline development and enable you to do regression testing. I’d advise samples are labelled with how they are captured from each source and capture stage as this makes them easy to understand and use.
📚If you’d like to read more, I’ve previously highlighted the importance of sample management here: https://knowledge.cribl.io/general-information-14/spring-cleaning-checklist-for-cribl-tips-for-a-review-1998
With your samples, you can test each pipeline and assess what’s happening. The pipeline diagnostics menu is an invaluable source of help here. Are fields being created or changed? Is there an event count difference caused by unrolls, drops, sampling or aggregation?
Steady state - Using Cribl Insights
Cribl Insights provides a great way of seeing trends across your data plane and is available in Cribl.Cloud. You can use Insights to track how source systems updates and pipeline updates in your Cribl deployment are impacting your data processing. Large changes in data volumes or shape are usually great triggers for a more in depth analysis. Has there been a protocol change, have there been any schema adjustments or is this just natural data volume changes?
You can use the alerting options to get particularly fine grained if you wish. For example, you may wish to set up alerts if the number of output fields for a critical destination ever changes away from a preset value. Alternatively, you can focus on the number of fields on a source and again, alert when there’s a variation in data shape. To avoid alert fatigue, I’d advise only applying this technique to brittle destinations whereby a schema mismatch may interrupt essential data flow.
📚To learn more about Cribl Insights, visit: https://docs.cribl.io/insights/about/
If you are thinking about monitoring your data estate more broadly, please do reach out to your Cribl team. There’s a lot of options available to focus attention on your most critical components and we periodically host monitoring Cribl presentations on this topic.
Change control testing and peer review
In an ideal deployment, any changes to data formats should be first tested outside of any production or live environment. Thanks to Cribl Packs, portability of pipelines and data flows has never been easier! To make the most of this architecture, I’d encourage you to think in a modular view of Packs with each being self contained using parameterisation and locally stored knowledge objects. This leads to a more portable experience.
📚If you want a full explanation of the importance of testing, please see this other article: https://knowledge.cribl.io/general-information-14/you-should-have-a-test-environment-2218
Finally, another important procedural control to consider is establishing peer reviews for proposed changes. Having a policy of having a M of N sign off limits the likelihood of an accidental change causing a larger error across your deployment. This can be facilitated using external orchestration tools such as Git branch controls.
Concluding remarks
Cribl enables unrivalled visibility and control into all aspects of your data plane. By understanding your flow objectives, using testing, continuously monitoring and adopting good procedural controls, your data can be exactly what you need it to be.