Resolved

Grafana dashboards not auto-deploying to Kubernetes cluster

Grafana
GitOps
FluxCD
Kustomize
Kubernetes
AKS
DevOps
Monitoring
CI/CD

Local Grafana dashboards required manual deployment and ran only on localhost | Migrated to GitOps workflow with FluxCD and Kustomize for automatic AKS deployment

Issue

The Grafana monitoring setup suffered from significant gaps in automation and consistency across environments. Dashboards were only available in local development environments (localhost:3000) and lacked an automated deployment pipeline to the production Azure Kubernetes Service (AKS) cluster. This led to configuration drift, manual deployment overhead, and inconsistent monitoring configurations across team members.

Culprit

The existing architecture relied on manual, unmanaged processes for dashboard deployment and configuration, including:

  1. Local Grafana provisioning configurations stored in a /grafana/ directory.
  2. Unmanaged dashboard files in a /dashboards/ directory without integration into deployment workflows.
  3. Manual kubectl port-forward access patterns for dashboard management.
  4. No connection to the GitOps pipeline, such as FluxCD or Kustomize, leaving deployments unmanaged.

This approach created technical debt in the form of orphaned configuration files, hardcoded local references, and no Kubernetes-native deployment manifests or ConfigMap generation.

Solution

The integration of Grafana with GitOps workflows was achieved through the following steps:

  1. Directory Restructure: All dashboard assets were moved to a Kubernetes-native location (k8s/infrastructure/monitoring/dashboards/).

  2. Kustomize ConfigMap Generation: ConfigMaps were generated for Grafana dashboards using Kustomize, enabling automatic deployment. Dashboards were labeled to trigger auto-discovery by Grafana’s sidecar component.

  3. HelmRelease Sidecar Configuration: Grafana was configured to use its sidecar feature to load dashboards automatically from labeled ConfigMaps. This eliminated the need for manual dashboard imports.

  4. Technical Debt Removal: Legacy configurations were cleaned up, including Grafana and dashboard directories, to ensure a streamlined and version-controlled deployment process.

  5. Deployment Flow Implementation: A GitOps pipeline was established, where dashboard updates would be deployed automatically within 30-60 seconds of a git push. The pipeline ensured configuration consistency across environments and reduced management overhead to zero-touch deployment.

These changes improved security by eliminating local credential storage and preparing the system for integration with Azure Key Vault and Azure AD authentication. The solution now supports five custom dashboards for system health, RAG performance, security compliance, business metrics, and MCP overview, ensuring comprehensive monitoring capabilities.