Kubernetes services only accessible via kubectl port-forward on localhost

Issue

Services in a k3s Kubernetes cluster, including CloudBeaver (database GUI), Prometheus (metrics), and Grafana (dashboards), were configured as ClusterIP by default. This restricted access to these services, requiring manual kubectl port-forward commands executed on the server. This setup imposed several operational limitations:

1. SSH Dependency: Required SSH access to the k3s control plane node to execute commands.
2. Terminal Session Requirement: Needed active terminal sessions for each service, with multiple port-forward commands running simultaneously.
3. Localhost Limitation: Access was restricted to the localhost of the server, preventing access from devices on the same local network.
4. Connection Instability: Connections dropped when SSH sessions terminated or laptops entered sleep mode.
5. Mobile/Remote Access: Impossible to access services from mobile devices or other computers on the local network.

The core limitation stemmed from the ClusterIP service type, which creates internal virtual IPs accessible only within the Kubernetes cluster, necessitating manual port forwarding for external access.

Culprit

The root cause was the default ClusterIP service configuration in Kubernetes manifests. This configuration:

• Created internal-only services with private IP addresses (e.g., 10.43.x.x), accessible solely within the cluster.
• Required manual execution of kubectl port-forward svc/<service-name> <local-port>:<service-port> commands for external access.
• Introduced network isolation, preventing direct LAN access to the internal ClusterIP.

The network architecture lacked a mechanism to expose services externally, resulting in a reliance on manual, session-dependent port forwarding.

Solution

Service Type Change

The solution involved switching the service type from ClusterIP to NodePort. This modification exposed services directly on the Kubernetes node’s IP address using a predefined port range (30000–32767). The updated configuration included:

• Internal Port: The original port used within the cluster.
• Target Port: The port on which the container listens.
• NodePort: A static port assigned on the node’s IP, accessible from the LAN.

Port Assignment Strategy

A structured port assignment strategy was implemented to ensure memorable and non-conflicting port mappings:

• CloudBeaver: Internal port 8978 → NodePort 30978
• Prometheus: Internal port 9090 → NodePort 30909
• Grafana: Internal port 3000 → NodePort 30300

This strategy used a base of 30000 plus the internal port number for intuitive mapping.

Key Implementation Steps

1. Service Manifest Updates: Modified Kubernetes manifests to specify the NodePort type and assigned explicit node ports.
2. Verification: Confirmed the updated service configurations using kubectl get svc, verifying the mapping of internal ports to node ports.
3. Connectivity Testing: Validated external access by connecting to the node’s IP using the assigned NodePort, confirming successful HTTP responses.

Resulting Architecture

The updated architecture enabled direct access from any device on the local network (LAN) to the Kubernetes services via the node’s IP and assigned NodePort. Key components included:

• Node IP: The physical IP address of the Kubernetes node, accessible from the LAN.
• NodePort: Ports in the 30000–32767 range, mapped to internal cluster services.
• kube-proxy: Managed iptables rules to forward traffic from NodePort to the appropriate pod.

This setup eliminated the need for SSH access, manual port forwarding, and terminal sessions while providing seamless LAN access.

Security Considerations

1. Network Scope: Limited to the local subnet (e.g., [IP]/24) to restrict access to trusted devices.
2. Authentication: Services retained their existing authentication mechanisms (e.g., Grafana credentials, CloudBeaver admin users).
3. Firewall: No additional firewall rules were required, as access was confined to the trusted LAN.
4. Port Range: Non-standard high ports (30000+) were used to minimize conflicts with other services.

Access Overview

With the NodePort configuration, services became accessible directly via their respective node ports on the Kubernetes node’s IP address, enabling:

• CloudBeaver: Accessible via [Node IP]:30978
• Prometheus: Accessible via [Node IP]:30909
• Grafana: Accessible via [Node IP]:30300

This solution resolved the operational limitations of the original ClusterIP setup, providing a seamless and scalable approach to exposing Kubernetes services on the local network.