Resolved

Resolving Redis connection errors in Kubernetes deployments

Redis
Kubernetes
DevOps
Networking
Debugging
Configuration
Backend

API pods crashed repeatedly due to Redis connection resets caused by a security feature in Redis that blocked internal cluster traffic. The issue was resolved by adjusting Redis configuration to align with the Kubernetes network policies.

A Kubernetes deployment of API pods (Python-based) experienced repeated crashes due to Redis connection errors (Connection reset by peer). Despite proper network connectivity and internal Kubernetes policies, the pods failed to establish connections with Redis, leading to CrashLoopBackOff and continuous restarts.

Issue

API pods repeatedly crashed and restarted, unable to establish connections to Redis. The error (Connection reset by peer) occurred despite:

  • Redis responding to internal PING commands.
  • Confirmed DNS resolution and service reachability.
  • No external firewall or security group changes.

This issue disrupted the deployment, causing instability and operational downtime.

Culprit

The root cause was Redis’s protected mode, which was enabled by default and blocked all non-localhost connections. Specifically:

  • Redis protected-mode was set to yes, which restricts connections to localhost (127.0.0.1) only.
  • The configuration lacked authentication (requirepass), forcing Redis to reject all pod-to-pod traffic within the cluster.
  • This behavior overrode Kubernetes network policies, actively resetting connections from non-localhost sources.

The Kubernetes cluster’s internal networking (e.g., pod-to-pod communication via ClusterIP) was incorrectly treated as “external” by Redis, causing connection failures.

Solution

Immediate Fix

Temporarily disabled protected mode to stabilize the deployment:

  • Executed a command in the Redis pod to disable protected mode.
  • This allowed API pods to establish connections immediately upon restart.

Permanent Fix

Updated the Redis deployment configuration to explicitly disable protected mode:

  • Modified the deployment manifest to include the --protected-mode no flag.
  • This ensures that Redis accepts connections from all cluster-internal pods without requiring authentication.

Security Considerations

Disabling protected mode was deemed safe in this context because:

  • Network isolation: Redis is deployed within a dedicated Kubernetes namespace with strict network policies restricting access to internal pods only.
  • No external exposure: Redis is accessible only via ClusterIP and not exposed externally via NodePort, LoadBalancer, or Ingress.
  • Equivalent protection: Kubernetes network policies provide the same security boundaries as Redis’s protected mode, making additional application-layer restrictions unnecessary.

Verification

After applying the fix:

  • Confirmed that protected mode was disabled persistently.
  • Validated that API pods could successfully connect to Redis.
  • Ensured no external access to Redis was possible.

This solution resolved the connection issues while maintaining security through Kubernetes-native networking controls.