Resolved

Kubernetes NetworkPolicy blocks Hugging Face dataset downloads

Kubernetes
NetworkPolicy
Security
DevOps
Containerization
Networking
Hugging Face
CI/CD

Default-deny-all NetworkPolicy in mcp-system namespace prevents pods from accessing external HTTPS endpoints | Results in ConnectionError when downloading datasets from Hugging Face Hub

The project addressed a Kubernetes NetworkPolicy issue in a security-hardened cluster where a default-deny-all policy blocked all pod egress traffic, causing a ConnectionError when attempting to download datasets from Hugging Face Hub.

Issue

A job responsible for indexing Bloomberg financial news datasets failed with a ConnectionError, specifically indicating that it couldn’t reach the Hugging Face Hub endpoint. This occurred despite valid credentials and endpoint availability, suggesting a network-level blockage.

Culprit

The cluster implemented a strict default-deny-all NetworkPolicy in the mcp-system namespace, which blocked all egress traffic by default. This policy required explicit allowances for any external communication. Since the Bloomberg indexing job lacked a corresponding NetworkPolicy, all external requests were dropped, resulting in the connection failure.

Solution

NetworkPolicy Configuration

To resolve the issue, a targeted NetworkPolicy was created to explicitly allow the necessary egress traffic for the Bloomberg indexing job:

  • DNS Access: Allowed UDP/TCP traffic on port 53 to the kube-system namespace for DNS resolution.
  • HTTPS/HTTP Egress: Allowed TCP traffic on ports 443 and 80 for accessing external endpoints, such as Hugging Face and Azure OpenAI.

Key Implementation Details

  • Pod Selector Matching: The NetworkPolicy targeted pods with the label app: bloomberg-indexing.
  • Minimal Privilege: Only the necessary ports were allowed, adhering to the principle of least privilege.
  • Verification: Testing was conducted using a debug pod to confirm successful DNS resolution and HTTPS connectivity.

Security Considerations

While the solution uses broad selectors for simplicity, production environments should consider:

  • Restricting access to specific CIDR blocks when possible.
  • Using egress gateways for audit trails.
  • Implementing network policy generators to maintain consistency across policies.

This approach resolved the connection issue, enabling successful dataset downloads from Hugging Face Hub while maintaining the cluster’s security posture of default-deny with explicit allowances.