Skip to main content
Question

Recurring UI problem with one worker group. Leader restart helps temporarily.

  • March 11, 2025
  • 19 replies
  • 47 views

Eugene Katz

We are seeing this issue quite often, the leader UI shows the the rotating wheel whenever a particular worker group is selected and it never loads the pipelines or routes. This issue happens only for one worker group and others load fine. A restart of the leader fixes the problem temporarily. I noticed the following error on leader but not clear on what it is indicating. The CPU usage of the leader is below average.

19 replies

Eugene Katz
  • Author
  • Employee
  • March 11, 2025

Is leader node trying to connect to the worker and it is not able to? is that what the message mean?


  • Employee
  • March 11, 2025

<@U0110PHRCSX&gt; What version of Cribl Stream are you running?


Eugene Katz
  • Author
  • Employee
  • March 11, 2025

4.1.2


  • Employee
  • March 11, 2025

How many worker process are connecting to the leader node?


  • Employee
  • March 11, 2025

And what is the size (cpu and ram) of the leader node?


Eugene Katz
  • Author
  • Employee
  • March 11, 2025

leader has 8vcpus and 16GB RAM


Eugene Katz
  • Author
  • Employee
  • March 11, 2025

A total of 184 worker processes


Eugene Katz
  • Author
  • Employee
  • March 11, 2025

but the problematic worker group only has 6 worker processes


only the API process from each node connects to the leader. RPC comms will run over that connection.


  • Employee
  • March 11, 2025

This is known issue with the RPC process in 4.1.1 and 4.1.2. please upgrade to 4.1.3


do you have 184 processes or nodes <@U0110PHRCSX&gt; ?


ok, i wanted to ensure we're using proper terms


Eugene Katz
  • Author
  • Employee
  • March 11, 2025

worker processes, we have a total of 6 workers


are you using collectors or pull-based sources like Office365 or Kinesis Streams?


Eugene Katz
  • Author
  • Employee
  • March 11, 2025

i will get the upgrade done and monitor


Eugene Katz
  • Author
  • Employee
  • March 11, 2025

<@U012ZP93EER&gt; sorry for the delay in responding i had to step out for a while


Eugene Katz
  • Author
  • Employee
  • March 11, 2025

yes, there are REST API collectors


ok, that increases the likelihood of encountering the known issue that Eric mentioned so upgrading is highly recommended like he said.


Eugene Katz
  • Author
  • Employee
  • March 11, 2025

ok, thank you