Deep Dive: Leveraging Projects for Small-Scale Self-Service

The best approach for self-service is generally to leverage the Large Scale advice above. However, if you are in a relatively small environment, you can also approach self-service by leveraging Projects. Projects within Cribl are isolated configuration spaces that allow for Cribl administrators to give permission to specific teams to configure their data processing pipelines without impacting the other data sources flowing through the platform. 

A quote from our docs page: “Projects enable this filtering and isolation. Each team gets secure access to its own sets of data, and each team’s data transformations and configuration changes don’t affect the other team’s data or configs. Additionally, data capture and samples are isolated to each project.” 

The documentation for projects outlines how to set them up in detail, but a quick overview of the pieces:

• Within a worker group, Cribl administrators can establish projects. Projects are an isolated configuration capability that leverages subscriptions for their incoming data.
• To get data to these projects, Cribl administrators set up subscriptions. These subscriptions are data filters that are prepared to send via projects. Subscriptions must be set up in order for projects to be leveraged.
• Teams must be assigned to the Projects in order to be able to edit them appropriately. 
• Once the projects are set up and configurations are committed by the Project Teams, the Cribl administrator or Cribl maintainer will still need to deploy the changes to the environment.

Performance Notes

Projects work by duplicating data and sending it into a segregated environment. This does add overhead to the data processing. Keep this in mind when scaling your worker groups for Projects.

Also, make sure you’re setting up Projects on a dedicated worker group that has room for mistakes; separate your mission-critical data from your Projects. Projects are great for application owners to own and manipulate their data, but if one of your users makes pipeline changes that impact processing severely, you want to make sure that doesn’t impact your other critical data.

RBAC Setup

To properly leverage projects and set your Cribl instance up for self-service, you’ll need to have SSO enabled in the tenant and leverage Teams. The documentation thoroughly describes the steps necessary to enable this, but here’s a quick summary of the Cribl configuration. All of these need to be done by an administrator of Cribl.

1. Under Organization -> Members & Teams, ensure that there is a Team created specifically for each project. These can be mapped to SSO groups for maximum efficiency. The Teams need User permissions to both the applicable Workspace and Stream. 
2. Under Settings -> Worker Groups (or Worker Group Settings), ensure that the same team is assigned User permissions to the Worker Group where the Project resides. 
3. Under the applicable Project, ensure that the same team is assigned Editor permissions under Members & Teams for that Project. 

Video Walkthrough

I did a quick tutorial on how to implement the above and that is available here: https://www.loom.com/share/6f54d58538234938843ff2ed3fbff125?sid=43be07aa-e10d-4ca6-8b1d-e5addd0b0588

Organization and SSO Integration

Now that you understand the Cribl side of the equation, how do you set this up in a self-service model? This is going to fluctuate depending on your organization’s tools and capabilities, but the idea here is to minimize the human element. Can you create a form where users sign up to be a part of a team, and it automatically gets routed to the team lead for approval? Can you leverage your organizational chart to automatically assign users to AD groups and map them to Cribl Teams? Think about how you want your Projects to be segregated, and design your Teams to match. Don’t forget that users can be a part of multiple Teams and Projects if needed. Put thought into these designs and how data can most efficiently be managed by your users; then give them the reins to own and manipulate their data as needed.

Example Scenario: 

Let’s say that you are ingesting data from Palo Alto Firewalls, Microsoft Defender, and a custom web application called Shine. Each of these log sources has a different team managing them. They know their data inside and out, and they know what data is relevant to the security team. Your SIEM cost is too high, and you need to focus on reduction without introducing risk, so you decide that your log source owners need to be involved in the logging process, but you don’t want any of the data to be available to other users nor do you want these log source owners to be able to make sweeping changes to the Cribl infrastructure. This is a perfect setup for projects. So how do you approach it? 

1. Choose your Projects design: You have a very obvious design in mind: one project for each log source. One team for each project. Easy enough. 
2. Choose your self-service design: This is an Azure shop; let’s use a Form that automatically places you in the correct group once approved by leadership.
   1. Create an Azure Form that asks for the Data Source Name and Justification
   2. Route these forms to the team lead of each team for approval
   3. Once approved, have an automation that adds that user to the appropriate group in Azure
3. Set up RBAC: Ensure that the teams associated with the forms from step 2 are mapped in Cribl SSO. Then make sure the permissions are set up as described in RBAC Setup. 
4. Create the applicable subscriptions: 
   1. Subscription with a filter for Palo Alto input
   2. Subscription with a filter for Microsoft Defender input
   3. Subscription with a filter for Shine input
5. Create the applicable projects: 
   1. Palo Alto Project
   2. Microsoft Defender Project
   3. Shine Project
6. Apply the appropriate group permissions to the Projects:
   1. Editor: Palo Alto Group -> Palo Alto Project
   2. Editor: Microsoft Defender Group -> Microsoft Defender Project
   3. Editor: Shine Group -> Shine Project
7. Have the members of the different groups test their access. Ensure each project is editable by the appropriate teams. They will be able to commit their changes, but they will NOT be able to deploy their changes. The Cribl administrators get final say on what gets deployed to the worker groups.