Resolved

LLM inference service lacks secure shared GPU access and usage tracking

Python
AI/ML
LLM
API
Kubernetes
DevOps
Security
Performance
vLLM
Redis

Direct GPU access for LLM inference creates security risks and lacks usage monitoring | Solution implements authenticated vLLM gateway with OpenAI-compatible API and Redis tracking

Issue

The auto-video-gen project required GPU-accelerated language model inference but faced challenges with direct GPU access, including:

  1. Security risks: Direct access without authentication posed a security threat.
  2. Lack of usage tracking: No mechanism existed to monitor token consumption or request volumes.
  3. Resource contention: Multiple services accessed shared GPUs without coordination, leading to inefficiencies.

The system needed to provide OpenAI API compatibility while implementing enterprise-grade security and usage controls for a production environment.

Culprit

The original architecture used direct connections to the vLLM service without:

  • Authentication validation (e.g., API keys).
  • Centralized usage tracking (missing Redis integration).
  • Request routing or load balancing.
  • Version control for model configurations.

Configuration files lacked essential security and routing settings, such as API keys and base URLs, which exposed direct access to the vLLM service.

Solution

To address these issues, an authenticated gateway architecture was implemented:

Authenticated Gateway

A three-tier system was established:

  1. FastAgent Client → vLLM Gateway (port 8001) → vLLM Service (port 8000).
  2. Authentication: The gateway validates API keys via the X-API-Key header against Kubernetes secrets.
  3. Request Proxies: All requests are routed through a single entry point with integrated Redis tracking.

Configuration Updates

  • Updated fastagent.config.yaml to specify models in a standardized format (e.g., vllm.Qwen/Qwen3-8B-FP8).
  • Added authentication details in fastagent.secrets.yaml, including:
    • API keys for secure access.
    • Base URLs for routing through the gateway.

Key Technical Features

  • OpenAI-Compatible Endpoint: Provides /v1/chat/completions to maintain compatibility with existing workflows.
  • Redis Usage Tracking: Monitors token consumption and request volumes, enabling usage analytics via endpoints like /usage/name.
  • FP8 Quantization: Supports FP8 precision for models like Qwen3-8B to optimize performance.
  • Low Latency: Achieves <1ms latency overhead with comprehensive health checks.

Deployment Flexibility

The gateway supports both:

  • Cluster-internal access: Uses Kubernetes DNS for internal routing.
  • Local development: Enables port forwarding for local testing.

Model Override Capability

Ensures backward compatibility by supporting model overrides through command-line arguments:

python agent.py --model=anthropic.claude-3-5-sonnet-20241022

This solution ensures secure, tracked, and efficient access to GPU-accelerated language models while maintaining compatibility with existing systems.