Resolved

MCP server crashes with PermissionError on audit log directory

Python
Docker
Kubernetes
Azure AKS
Permissions
DevOps
Logging
Security

Azure AKS container fails with Permission denied when writing audit logs to /var/log/mcp-server | Non-root appuser (UID 1000) cannot access system log directory in production Docker image

The project addressed a permission error in a Python-based MCP RAG SDK deployed on Azure Kubernetes Service (AKS), where the server container was unable to write audit logs due to restricted access to the system log directory. The issue prevented the application from starting and required a solution to ensure the container’s non-root user had the necessary permissions.

Issue

The MCP server container crashed during startup with a PermissionError, specifically [Errno 13] Permission denied: '/var/log/mcp-server/audit.jsonl'. This occurred when the application attempted to initialize an audit logger, resulting in consistent crashes across pod restarts in the production AKS environment, despite successful image builds and Kubernetes deployments.

Culprit

The issue stemmed from hardcoded log paths in the legacy codebase, specifically:

  • /src/monitoring/audit.py (line 108), where the log_path parameter in AuditLogger.__init__() was set to /var/log/mcp-server/audit.jsonl.
  • /src/server.py (line 77), where the audit_log_path field was set to the same path.

The /var/log/mcp-server/ directory required root privileges to access, and the container’s non-root appuser (UID 1000) lacked the necessary permissions. Additionally, the Dockerfile used the legacy /src/ codebase instead of the newer /rag-backend/src/ directory, which contained an updated implementation.

Solution

Path Configuration Fixes

The solution involved updating the hardcoded paths to use a writable directory within the container:

  • Modified the log_path in audit.py to point to /app/logs/audit.jsonl.
  • Updated the audit_log_path field in server.py to the same directory.

Image Reconstruction

  1. Rebuilt the production Docker image with the updated paths to ensure the changes were included.
  2. Pushed the new image to the container registry.

Deployment Verification

  1. Forced a pod restart to pull the updated image and verify that the changes resolved the permission issue.
  2. Confirmed successful operation through pod status checks, audit log verification, and the operational status of the GitOps stack (FluxCD, Prometheus, Grafana).

Key Technical Insight

The issue revealed an architectural complexity with dual-codebase maintenance, where:

  • The legacy /src/ directory contained the production implementation used by the Dockerfile.
  • The newer /rag-backend/src/ directory contained an updated but unused implementation.

This discrepancy created maintenance challenges and necessitated explicit documentation for future developers to ensure all changes were made in the correct location.