Kubernetes pod crash due to permission issues writing audit logs

Issue

A Kubernetes pod running the MCP server crashed with a PermissionError when attempting to write audit logs to /var/log/mcp-server. The error occurred because the container ran as a non-root user (UID 1000) without write permissions to the host’s /var/log directory, despite the existence of a writable emptyDir volume at /app/logs.

Culprit

The issue stemmed from:

1. Hardcoded log path: The application targeted /var/log/mcp-server, a directory owned by root:root with 755 permissions, which is inaccessible to non-root users.
2. Unused volume: While an emptyDir volume was mounted at /app/logs with read-write permissions, the application was not configured to use this volume for logging.
3. Missing ConfigMap: There was no environment variable override in the ConfigMap to redirect the log path to the writable volume.

Solution

Redirect Log Path

Updated the ConfigMap to include an environment variable redirecting the log path to the writable emptyDir volume:

• Log Path: Changed from /var/log/mcp-server to /app/logs/audit.log to leverage the writable volume.
• Permissions: Utilized the emptyDir volume (default rw permissions) to ensure the non-root user could write logs.
• Security: Maintained the non-root security context (UID 1000) without privilege escalation.

This solution resolved the permission-related crashes, ensuring audit logs were written successfully and the pod remained stable while adhering to least-privilege security practices.