Troubleshooting Egress Connection Handshake Timeout Errors in Cribl on OpenShift / OKD

Objective

Diagnose and resolve egress "Connection Handshake Timeout" errors affecting Cribl workloads running on OpenShift / OKD. These errors indicate TLS/SSL negotiation failures that can exist at multiple network layers—from pod-level policies to cluster infrastructure. This guide follows a bottom-up diagnostic approach, starting at the pod and expanding outward to cluster components.

Environment

• Cribl Stream or Cribl Edge (any version)
• Deployment type: Kubernetes / OpenShift (OKD)
• OpenShift Container Platform 4.x or OKD equivalent
• Network plugin: OVN-Kubernetes or OpenShift SDN

Procedure

Step 1: Pod-Level Diagnostics

Determine whether the issue affects a single pod or multiple applications.

1.1 Test Connectivity and DNS

1. Open a shell in the affected pod:
   oc exec -it <pod_name> -- /bin/bash # Use /bin/sh if bash is unavailable

2. Test raw network connectivity:
   ping 8.8.8.8
   Failure indicates a foundational block (egress firewall or node network issue) rather than a DNS or application problem.

3. Test DNS resolution:
   nslookup cribl.io
   Failure points to DNS misconfiguration in the pod's resolv.conf, nsswitch.conf, or the cluster DNS service.

4. Simulate the connection with verbose TLS output:
   curl -v https://cdn.cribl.io
   The -v flag reveals the exact failure point in the TLS handshake. Run with -k to bypass certificate validation — if this succeeds, TLS interception or certificate trust is the root issue.

1.2 Check Network Policies

Misconfigured NetworkPolicy objects are a common cause. Many projects use default-deny policies that block unwhitelisted egress.

5. List policies in the namespace:
   oc get networkpolicy -n <project_name>

6. Inspect policy rules:
   oc describe networkpolicy <policy_name> -n <project_name>

7. Enable ACL audit logging to capture deny decisions (OVN-Kubernetes clusters only):
   oc annotate namespace <namespace> \ k8s.ovn.org/acl-logging='{"allow": "info", "deny": "info"}'
   View logs in ovn-controller pods on each node.

Note: This annotation generates high log volume — remove it after capturing the relevant entries.

Step 2: Cluster-Level Networking

If pod-level checks pass, investigate the OpenShift networking infrastructure.

2.1 SDN and Egress Firewall

8. Check for an EgressNetworkPolicy (restricts outbound traffic at the project level):
   oc get egressnetworkpolicy -n <project_name>
   A broad Deny rule can inadvertently block intended Allow rules. Review rule ordering carefully.

9. Verify HostSubnet configuration:
   oc get hostsubnets

Confirm each node has a valid HostSubnet and that egress CIDRs are assigned correctly if using egress IPs.

2.2 DNS Verification

10. Check DNS operator health:

    oc get clusteroperator dns

11. Review DNS pod logs:

    oc logs -n openshift-dns -l dns.operator.openshift.io/daemonset-dns=default

12. Inside the pod, verify that resolv.conf points to the cluster DNS service IP:

    cat /etc/resolv.conf

Step 3: External Factors and Advanced Configuration

If the issue persists, examine proxy configuration, certificate trust, and egress IP settings.

3.1 Corporate Proxy Configuration

13. Verify the proxy environment variables are set in the pod's deployment spec:

    • HTTP_PROXY

    • HTTPS_PROXY

    • NO_PROXY

14. Test proxy reachability from the pod:

    nc -vz <proxy_host> <proxy_port>

3.2 Certificate Issues

Handshake timeouts often stem from TLS certificate problems:

15. Debug with OpenSSL to reveal the certificate chain:

    openssl s_client -connect <hostname>:<port>

    Review the output for trust failures or missing intermediate certificates.

3.3 Egress IP and Router Configuration

Egress IPs provide a consistent source IP for external firewall rules.

16. Check the namespace egress IP assignment:

    oc get netnamespace <project_name> -o yaml

17. If an egress IP is configured:

    • Verify the hosting node has external connectivity.

    • Confirm that external firewalls allow traffic from this source IP.

18. For dedicated egress router deployments, review egress router logs for connection or routing errors.

Quick Reference

Last Validated

Cribl Stream / Cribl Edge — 2026-04-29 (OpenShift Container Platform 4.x)

Additional Information

• OpenShift Network Policy Documentation
• Cribl Docs: Deploying on Kubernetes