Resolved

rule Gatekeeper policies reject valid workloads on small Kubernetes nodes

Kubernetes
rule
Gatekeeper
Policy
Azure
AKS
DevOps
Security

Gatekeeper admission policies with hardcoded resource limits (4 CPU, 8GB) failed on B2s nodes (2 CPU, 4GB), while mandatory labels blocked development environments. Policies were customized to align with node capacity and allow development flexibility.

Issue

Gatekeeper admission controller rejected workloads due to two critical failures:

  1. Resource Limits: Policies enforced limits of 4 CPU cores and 8GB RAM, which exceeded the capacity of B2s nodes (2 CPU cores, 4GB RAM).
  2. Label Requirements: Mandatory cost-center label validation blocked development environments that lacked formal cost tracking labels.

Error examples included:

  • CPU/memory limits exceeding node capacity.
  • Missing required labels in development namespaces.

Culprit

Resource Mismatch

The policies were designed for enterprise-grade nodes with higher resource capacity, but the B2s node (2 vCPU, 4GB RAM) could not accommodate the hardcoded limits of 4 CPU cores and 8GB RAM. This mismatch caused all workloads to be rejected due to insufficient resources.

Development Blockers

Strict label validation enforced a formal cost-center format (e.g., ^[A-Z]{2,4}-[0-9]{4}$) that was not applicable to development environments. This prevented testing and deployment in dev namespaces.

Solution

Resource Constraint Alignment

Adjusted the resource limits in Gatekeeper policies to match the B2s node capacity:

  • CPU Limit: Reduced from 4 cores to 2 cores (100% of node capacity).
  • Memory Limit: Reduced from 8GB to 2GB (50% of node capacity to allow system overhead).

Development-Friendly Labels

Made the cost-center label optional for development environments while retaining its mandatory status for production.

  • Development environments: Allowed to proceed without the label.
  • Production environments: Enforced the label requirement.

Validation and Deployment

  1. Policy Validation: Ensured all policies were syntactically correct.
  2. Deployment Readiness: Designed policies to be activated in a controlled manner.
  3. Testing: Validated policies in a Kubernetes cluster with Gatekeeper installed.

Impact:

  • Resource Utilization: Achieved 100% utilization of node capacity.
  • Security: Maintained image signing requirements.
  • Developer Experience: Unblocked development workflows while preserving production-grade enforcement.

This solution balanced security and flexibility, ensuring compliance with resource constraints while accommodating development needs.