Resolving configuration loss in a Kubernetes-deployed database management tool

A database management tool deployed in Kubernetes experienced repeated configuration loss after pod restarts, requiring manual reconfiguration every ~10 hours. The issue stemmed from the tool’s reliance on an embedded database file that was not persisted across restarts, causing connections, user credentials, and workspace settings to be lost.

Issue

The tool threw a configuration expiration error every ~10 hours, and all configurations (e.g., admin users, database connections) were lost after pod restarts. Key symptoms included:

• Configuration timer reset: The tool treated each restart as a fresh instance, enforcing a 10-hour limit.
• Lost embedded database: The tool stored configurations in a local database file (H2 database) that was not persisted.
• Manual reconfiguration: Admins had to recreate user credentials and connections after every restart.

Culprit

The root cause was the lack of persistent storage for the tool’s configuration files. The tool relied on:

• An embedded database (H2) storing configurations in a non-persistent directory (/opt/cloudbeaver/workspace).
• Kubernetes ephemeral storage for the pod, which wiped configurations on restarts.
• A security feature that enforced a 10-hour timer for unconfigured instances.

Solution

To resolve the issue, a PersistentVolumeClaim (PVC) was implemented to retain configurations across pod restarts. This involved:

1. Configuring a PersistentVolumeClaim

• Allocated a 1GB PVC using a storage class (e.g., local-path) to store configurations.
• Mounted the PVC to the tool’s configuration directory (/opt/cloudbeaver/workspace).

2. Updating Deployment Configuration

• Added a volume mount in the Kubernetes deployment to ensure the PVC was accessible.
• Ensured the embedded database file (H2 database) was stored in the mounted volume.

3. Verification

• Pod restart test: Deleted the pod to simulate a restart and confirmed configurations persisted.
• Database connection retention: Verified that PostgreSQL and other database connections remained intact.

Key Outcomes

• Zero configuration loss: Persistent storage retained all settings across restarts.
• Reduced setup time: Eliminated manual reconfiguration after restarts.
• Improved reliability: Ensured consistent access to database connections and user credentials.

This solution resolved the configuration expiration issue while ensuring robustness in Kubernetes deployments.