We are seeing this issue quite often, the leader UI shows the the rotating wheel whenever a particular worker group is selected and it never loads the pipelines or routes. This issue happens only for one worker group and others load fine. A restart of the leader fixes the problem temporarily. I noticed the following error on leader but not clear on what it is indicating. The CPU usage of the leader is below average.
Page 1 / 1
Is leader node trying to connect to the worker and it is not able to? is that what the message mean?
<@U0110PHRCSX> What version of Cribl Stream are you running?
4.1.2
How many worker process are connecting to the leader node?
And what is the size (cpu and ram) of the leader node?
leader has 8vcpus and 16GB RAM
A total of 184 worker processes
but the problematic worker group only has 6 worker processes
only the API process from each node connects to the leader. RPC comms will run over that connection.
This is known issue with the RPC process in 4.1.1 and 4.1.2. please upgrade to 4.1.3
do you have 184 processes or nodes <@U0110PHRCSX> ?
ok, i wanted to ensure we're using proper terms
worker processes, we have a total of 6 workers
are you using collectors or pull-based sources like Office365 or Kinesis Streams?
i will get the upgrade done and monitor
<@U012ZP93EER> sorry for the delay in responding i had to step out for a while
yes, there are REST API collectors
ok, that increases the likelihood of encountering the known issue that Eric mentioned so upgrading is highly recommended like he said.
ok, thank you
Reply
Login to the community
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.