Terraform deployment blocked by circular dependencies in nested modules

Working with Terraform on Azure to deploy a cost-optimized MCP RAG SDK infrastructure while transitioning from a banking-grade architecture (~$1,500/month) to a $100/month constraint. The implementation uses GitOps (FluxCD), rule Gatekeeper policies, and hybrid search RAG architecture with Azure OpenAI embeddings.

Issue

Terraform deployment was completely blocked by circular dependency errors in the nested module architecture. The terraform plan command failed due to unresolved variable references in dynamic blocks, preventing any Azure resource provisioning. Despite 95% of application code being production-ready, zero infrastructure could be deployed due to these issues.

Key Symptoms:

1. Circular Dependencies: Modules referenced each other’s outputs before creation.
2. Dynamic Block Failures: Terraform couldn’t resolve variables in nested configurations.
3. Zero Deployment Progress: None of the 20+ Azure resources could be provisioned despite two months of development.
4. Budget Constraint Conflict: The original banking-grade architecture was incompatible with the $100/month requirement.

Culprit

Root Cause 1: Over-Engineered Module Structure

The original Terraform implementation used nested modules where dependencies between modules created circular references. For example, the network module depended on the security module, which in turn depended on the network module, creating a deadlock.

Root Cause 2: Budget Reality Mismatch

The original banking-grade architecture was significantly over-provisioned for the new budget constraints:

• AKS nodes: Reduced from 3x D4s_v3 ($600+/month) to 1x B2s spot ($15/month).
• KeyVault: Switched from Premium with HSM ($200+/month) to Standard.
• ACR: Switched from Premium with geo-replication ($150+/month) to Standard.
• Network Security: Replaced Azure Firewall ($1,000+/month) with NSG rules.
• Log Retention: Reduced from 90 days ($50+/month) to 30 days ($10/month).

Root Cause 3: Sequential Development Mistake

The team completed 2,800+ lines of application code and achieved 80% test coverage before validating infrastructure deployment, leading to a situation where no infrastructure could support the application.

Solution

Solution 1: Layer Separation Architecture

The nested modules were replaced with a flat, two-layer architecture:

• Layer 1: Foundation: Defined core infrastructure resources (e.g., VNet, KeyVault) in the root directory without using modules.
• Layer 2: Application: Used terraform_remote_state to reference outputs from the foundation layer, ensuring proper dependency management.

Solution 2: Cost-Optimized Component Selection

Significant cost reductions were achieved by switching to budget-friendly alternatives:

• AKS: Switched to spot instances and reduced node size.
• KeyVault: Used the standard tier.
• ACR: Standard tier without geo-replication.
• Network Security: NSG rules instead of Azure Firewall.
• Embeddings Model: Switched to a lower-cost model.

Solution 3: Infrastructure-First Development

1. Paused application development at 95% completion.
2. Refactored Terraform to use layer separation.
3. Validated deployment of the foundation layer first.
4. Implemented cost tracking with Azure Budget alerts.
5. Documented tradeoffs between banking-grade and budget constraints.

Verification Results

• Before refactor: Terraform plan failed due to circular dependencies.
• After refactor: Foundation layer deployed successfully (8 resources), followed by the application layer (12 resources).

Cost Impact

• Original cost: ~$1,500/month
• Current cost: ~$85/month
• Savings: 94% reduction in monthly costs.