Grafana dashboards not deployed to production AKS cluster

Working with kube-prometheus-stack (Prometheus Operator) in Azure Kubernetes Service (AKS) to migrate Grafana from a localhost development environment to a production GitOps deployment using FluxCD and Helm. The solution integrates custom dashboards into the Kubernetes monitoring stack with proper ConfigMaps and HelmRelease configurations.

Issue

Grafana was running only on localhost ([IP]:3000) with no production deployment in the AKS cluster. Four critical dashboards (system health, RAG performance, security/compliance, business metrics) existed as orphaned JSON files in the repository but were not auto-loaded into any production monitoring system. The repository contained three abandoned directories (/grafana/, /dashboards/, /config/grafana/), creating confusion between local development and production infrastructure.

Culprit

1. Architectural Misalignment: Grafana was deployed outside the Kubernetes cluster (localhost) instead of within AKS as part of the observability stack.
2. Missing GitOps Integration: Dashboard files existed but were not referenced in any HelmRelease or ConfigMap for Kubernetes deployment.
3. Repository Pollution: Orphaned configuration directories remained from previous local development attempts.
4. Configuration Gap: The existing kube-prometheus-stack HelmRelease lacked custom dashboard configurations.

Solution

GitOps Directory Restructuring

• Moved dashboards to a proper GitOps location and cleaned up technical debt by removing unused directories.

Dashboard ConfigMap Implementation

• Created a Kubernetes ConfigMap with auto-discovery labels to ensure dashboards are automatically loaded into Grafana.

HelmRelease Configuration

• Updated the HelmRelease configuration to include references to custom dashboards, ensuring they are deployed alongside the Grafana instance.

Production Deployment Flow

1. Git Push → FluxCD Sync: Changes triggered automated deployment through the GitOps pipeline.
2. Helm Release: The kube-prometheus-stack deployed with custom dashboards integrated.
3. Kubernetes Resources Created: Verified the deployment of Grafana, Prometheus, and Alertmanager pods, as well as the Grafana ConfigMap.
4. Access Methods: Provided both development (port-forwarding) and production (HTTPS ingress) access methods.

Security Implementation

• Implemented TLS termination and Azure AD integration for authentication.
• Enforced RBAC policies to restrict dashboard access.
• Configured audit logging for all administrative actions.

Result: Four production dashboards were auto-loaded into Grafana running within AKS, with a clean repository structure and proper GitOps management. The monitoring stack now includes:

• System health (latency, errors, circuit breakers)
• RAG performance (search, embeddings, cache hits)
• Security/compliance (authentication, RBAC, audit logs)
• Business metrics (token usage, costs, users).