Introduction

At this point, it feels like every team across an enterprise is sending data somewhere. Whether it be the SIEM, a data lake, or an analytics platform, data is flying all around. This puts a large burden on the team that is in charge of directing and processing that data in Cribl. However, there are a few possible paths forward that allow for data owners to take charge of their data without impacting the larger Cribl deployment: leveraging projects (small scale), leveraging worker groups (medium scale), and leveraging Workspaces and Git (large scale). 

Implementing at Scale

Adapting this thought process within an enterprise takes a lot of process and procedure to get right. Teams need to consider how they want their processes to look as they implement this. Considerations should include things like: 

• Who has the capability to review, commit, and deploy changes in Cribl Production?
• What standards do I want to implement for my data? Naming conventions? Required fields? Tagging? 
• Where does testing of the pipeline changes occur? 
• What does your development pipeline look like? 
• What tools do you have for change control in your environment? 
• What kind of access do you want data owners to have in Cribl? 

Data Governance

A lot of the time, we focus on technological solutions to our data problems, and a lot of the time that makes sense. However, when implementing a technology solution such as Cribl where you want to enable application owners to truly own their data without impacting the function of the tools in play, you have to consider the non-technical aspects of how this will look. This is where data governance comes in. This blog post does a great job of outlining the elements of data governance and how you could leverage Cribl for some of them. The scope of this blog post is for telemetry data, but the concepts can be applied to a larger audience. 

The main takeaway here is to consider the people and process part of the equation more than you would anticipate. Establish the standards, change control, access management, development process, and approval process before trying to implement the technological solution. 

Best Practices to Consider

A few best practices to keep in mind when you’re developing the standards and expectations in your organization: 

• Create a naming convention that indicates source, environment, and destination. Do this for every piece of the puzzle, including samples; you’ll regret saving things with random names if you let them stick around.
• Leverage comments in pipelines and routes to describe the goals or contents of each.
• Have a change control cadence to ensure all changes are being addressed and discussed in a timely manner.
• Publish internal documentation about your specific process for developing new integrations and promoting them into production.
• Define the data owners (team owning the application / sending the data) and data users (team receiving / leveraging / searching the data) for each data source and have expectations for each role.
• Discuss data requirements with the data users when developing pipelines; make sure that the data being sent meets the needs of the data users. For example: check with your security team about which fields they use for investigations and ensure those fields are properly maintained.
• Ensure new personnel take advantage of the Cribl University training in order to make sure they understand the work that they are doing; require completion of admin certification before granting access to any production environments.

Self-Service Approaches

As mentioned above, there are a few different ways of implementing a self-service model for Cribl within your organization. The scale of what you are trying to accomplish and the tools you have available should help determine the path you choose going forward. Below are some quick summaries of different approaches. Within each section, you’ll find a link to more technical details about the deployment process for each of them. Check them out and see what would work best for you.

Large Scale Self-Service via Workspaces and Git

For more information on this deployment approach, see the Deep Dive: Large Scale Self-Service via Workspaces and Git

What this enables

This approach empowers application/data teams to onboard and shape their own data by building a Cribl Pack in a development environment, while Cribl administrators maintain final approval and production quality. This scales onboarding without forcing admins to touch every data source.

Why it works

• Scalability: App teams do the heavy lifting; admins focus on guardrails and final gatekeeping.
• Quality and safety: Isolation plus reviews prevent bad changes from reaching production.
• Traceability: Git history and structured change control provide an audit trail end-to-end.

Core principles

• Keep control. Admins own production promotion. Use a change control system (e.g., JIRA/ServiceNow) to track scope, ownership, readiness items, and rationale; templates and forms reduce errors and missing info.
• Make Git the source of truth. Use Git to move packs across environments and track changes. Keep one repo per pack, with a dev and prod/main branch to clearly separate development from production history.
• Isolate development. Use Workspaces to give dev teams an isolated Cribl environment inside your Cribl.Cloud tenant with controlled access and no shared configs/data flows with production. Workspaces provide separate, isolated instances while sharing org-level controls. If Workspaces aren’t possible, teams can use the free on‑premises download to run Cribl locally for development.

Flow at a glance

1. Create the pack. In a dev Workspace (or local dev), create a pack and follow your governance standards (naming, tags, versioning).
2. Ingest data. Add sources and confirm data is flowing. If you can’t connect the live source, use samples or replay from object storage (e.g., S3) to bring representative data into dev for testing.  
3. Build pipelines. Application/log owners design pipelines (parse, keep needed fields, reshape/serialize) to meet downstream and cost goals, then validate with test data.
4. Set up destination (optional). Depending on your responsibility matrix, either the dev team or admins add the destination. Clearly mark data as development and document what to change for production.
5. Create the route. Wire source → pipeline → destination inside the pack.
6. Export to Git. Export the pack, place its contents at the root of a single Git repository, and commit to the dev branch. Update the change ticket to signal development complete.
7. Review and promote. Admins review in Git or the dev environment. If changes are needed, return to the team; if approved, merge to prod/main and import to production, update tags for production, deploy, test, and close the ticket.

Medium-Scale Self-Service with Worker Groups

For more information on this deployment approach, see the Deep Dive: Medium-Scale Self-Service with Worker Groups

What this enables

A dedicated Worker Group for development gives data owners a contained place to build pipelines and Packs, while admins maintain production control via RBAC, Git, and change management. Worker Groups isolate config (routes, pipelines, sources, destinations) and version control per group.

Why it works

• Scalability: Teams build and iterate autonomously in a well-defined playground.
• Safety: Isolation per Worker Group and admin-controlled deployment reduce risk to production.  
• Traceability: Git plus change control creates an auditable path from request to release.

Core principles

• Keep control. Admins retain final say on production promotion; use a lightweight but consistent change control process (e.g., JIRA/ServiceNow) to track ownership, readiness tasks, and rationale.
• Make Git the source of truth. Use Git to move Packs through dev → prod, with one repo per pack and dev/main branches to separate testing from production history.
• Use RBAC + Teams. Apply Permissions at the Worker Group level (e.g., Editor vs. Admin), and manage access via Teams mapped to your IdP/SSO groups for scalable onboarding and offboarding.

Leveraging Projects for Small-Scale Self-Service

For more information on this deployment approach, see the Deep Dive: Leveraging Projects for Small-Scale Self-Service

What this enables

Projects provide isolated spaces where specific teams can configure their own data flows and transformations without impacting other data or configurations, while admins retain control and scope access to only what each team needs. Data capture and samples are isolated per project.

Core pieces

• Projects sit within a Worker Group and tie together selected data, processing, destinations, and authorized users, giving editors a streamlined visual UI.  
• Subscriptions filter a subset of the Worker Group’s incoming data stream that Projects expose to editors; they function like Routes but operate in parallel and without backpressure to senders.  
• RBAC with Members & Teams controls access; admins share Projects with Teams/Members at the Project level (Editor, Maintainer, Read Only), with prerequisite User permissions at Org/Stream and Worker Group levels.  
• Requirements: Projects require an Enterprise plan or on‑prem license; Subscriptions ingest only from Sources set to “Send to Routes” (not QuickConnect).

Flow at a glance

1. Set up Subscriptions. Define filter expressions (and optional pre‑processing) to segment the Worker Group’s incoming data for each team’s scope.  
2. Create Projects. Associate the right Subscriptions and Destinations, grant Members/Teams access, and use the Project view to link Subscriptions → Pipelines/Packs → Destinations.  
3. Commit and deploy. Editors can commit Project changes to Git; admins maintain deployment control, and changes must be committed and deployed to activate data flow.  
4. Loom video: https://www.loom.com/share/6f54d58538234938843ff2ed3fbff125?sid=43be07aa-e10d-4ca6-8b1d-e5addd0b0588