Security and Configuration Fixes for Grafana Monitoring Stack

Issue

Two critical problems were identified in the Grafana/Prometheus/CloudBeaver monitoring stack:

1. Security vulnerability: Grafana admin credentials were hardcoded in the Kubernetes manifest (grafana-deployment.yaml), violating centralized credential management practices and exposing sensitive information in version control history.
2. Dashboard failure: The CloudBeaver monitoring dashboard failed to load due to a missing uid field in the Prometheus datasource configuration, causing Grafana to generate random UIDs and break dashboard references.

Culprit

Hardcoded Credentials

The Kubernetes manifest included literal credentials:

admin-password: name-grafana-2025

Despite the presence of a secure .env file containing the correct variable:

GRAFANA_ADMIN_PASSWORD=h3894JKpuhg843fDFdfHUddsoosFe

The variable was not properly exported in the script responsible for environment variable substitution (common.sh), causing envsubst to replace ${GRAFANA_ADMIN_PASSWORD} with an empty string. This left the Grafana admin password exposed in the manifest.

Datasource UID Mismatch

The dashboard configuration referenced a Prometheus datasource using:

"datasource": {"uid": "prometheus"}

However, the manifest lacked the corresponding uid field:

datasources:
  - name: Prometheus
    type: prometheus
    # Missing uid: prometheus

This mismatch caused Grafana to auto-generate random UIDs, breaking the dashboard references and preventing successful loading.

Solution

Environment Variable Integration

1. Exported variables in common.sh to ensure proper substitution:

export GRAFANA_ADMIN_PASSWORD GRAFANA_API_KEY GRAFANA_URL

2. **Updated the manifest** to use environment variable substitution:
```yaml
admin-password: ${GRAFANA_ADMIN_PASSWORD}

Datasource Configuration Fix

Added the explicit uid field to align with dashboard references:

datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: http://prometheus-service:9090

Verification Process

1. Environment verification:

   source k3s/scripts/common.sh && load_env_vars

echo “GRAFANA_ADMIN_PASSWORD is set: ${GRAFANA_ADMIN_PASSWORD:+YES}”

2. **Substitution test**:
```bash
envsubst < k3s/manifests/grafana-deployment.yaml | grep admin-password
# Expected Output: admin-password: h3894JKpuhg843fDFdfHUddsoosFe

3. Kubernetes verification:

   kubectl get secret grafana-admin -n name \
     -o jsonpath='{.data.admin-password}' | base64 -d
   # Expected Output: h3894JKpuhg843fDFdfHUddsoosFe

Result:

• Security compliance: Grafana credentials are now managed securely through environment variables, eliminating hardcoded values.
• Dashboard functionality: CloudBeaver and other dashboards load correctly with proper datasource UID alignment.
• Consistent deployment: All monitoring services (Grafana, Prometheus, CloudBeaver) deploy successfully via the pipeline with proper environment variable substitution.