I’ve made the switch from a single instance to a distributed setup (1 leader and 2 worker nodes). My old server, which receives logs from a variety of sources, is now acting as a worker. What happens if this worker goes down, because all of the sources are sending logs to one single worker IP? What should I do to guarantee appropriate high availability?
Question
Changing from single instance to distributed setup - high availability
One of the advantages of switching to a distributed architecture is the ability to run multiple worker instances. In fact, its best practice to run at least 2 for just the scenario you describe above. That way you can absorb one failure without interrupting service. Add a second instance, and configure your sources to load balance data sent to Stream, or send through a load balancer.
Hey,
In the past with clients we have utilised an NGINX instance to load balance across the two worker servers, per port (if applicable). I would like to note that this can be done with any decent LB, with a VIP attached. You could also perform DNS load balancing, however this would only even the load, not provide high availability.
This also depends on your data sources. For example Splunk UFs have the ability to automatically load balance across multiple IPs.
Hope that helps!
Sign up
Already have an account? Login
Login to the community
No account yet? Create an account
Using your Cribl Curious or University Account
User Login Employee loginEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.