Resolved

Prometheus cannot scrape API metrics with connection refused errors

Prometheus
Kubernetes
NetworkPolicy
Python
FastAPI
Monitoring
DevOps
Debugging

Prometheus targets show health down with connection refused errors despite running API services | Network policy blocking metric scraping combined with uninitialized Prometheus metrics at startup

Issue

The FastMCP monitoring stack experienced failures where Prometheus reported health: down with connection refusal errors when attempting to scrape metrics from the /metrics endpoint. Despite the endpoint being accessible and functional within the API pods, FastMCP-specific metrics (fastmcp_*) were not being populated. This resulted in:

  1. All API targets appearing as unhealthy in the Prometheus UI.
  2. Empty metric values for all FastMCP custom metrics.
  3. Manual curl requests to /metrics from within API pods succeeding.
  4. Failed wget attempts from Prometheus pods to the API endpoints.

Culprit

The monitoring failure was caused by two distinct technical issues:

Network Policy Blocking

The Kubernetes NetworkPolicy resource, designed to restrict ingress traffic to the API service, was overly restrictive. It only permitted traffic from cloudflare-tunnel pods and explicitly blocked Prometheus pods, which were labeled app: prometheus. This prevented Prometheus from scraping metrics, even though the /metrics endpoint was available.

Uninitialized Metrics

The FastMCP metrics system required explicit initialization by importing main_mcp_server.py. However, this import only occurred during email processing by the email worker, not during FastAPI startup. As a result, all fastmcp_* metrics remained unregistered with Prometheus, even though the /metrics endpoint was operational.

Solution

Network Policy Fix

The Kubernetes NetworkPolicy was updated to include Prometheus as an allowed source for ingress traffic. This ensured that Prometheus pods could successfully communicate with the API service and scrape metrics without being blocked.

Metrics Initialization

A startup event handler was added to the FastAPI application to explicitly initialize the FastMCP server. This initialization occurs during application startup, ensuring that all fastmcp_* metrics are registered with Prometheus. The handler also includes error handling to log any issues during initialization, ensuring that the system remains operational even if metrics are temporarily unavailable.

Verification Implementation

  1. Static Metrics Validation: Verified the registration of metrics by querying the Prometheus API for metrics related to server counts and versions, ensuring they were correctly populated.

  2. Dynamic Metrics Framework: Established a framework for collecting 15 custom FastMCP metrics, including:

    • Tool Call Rates: Metrics to track the frequency of tool calls.
    • Execution Durations: Metrics to measure the duration of tool executions.
    • Email Workflow Tracking: Metrics to monitor the status and progress of email workflows.
    • Quality Ratings: Metrics to assess the quality of responses generated by the system.
    • Duplicate Detection: Metrics to track the number of duplicate requests blocked.

  3. Grafana Integration: Created a dedicated Grafana dashboard with 11 panels to visualize the collected metrics. This dashboard was configured to persist using a ConfigMap, ensuring it remains available across deployments.

The solution now enables complete monitoring of the FastMCP system, providing visibility into both static infrastructure metrics and dynamic workflow metrics that populate during email processing. The monitoring architecture follows a clear flow: FastAPI → Prometheus Middleware → Metrics Endpoint → Prometheus Scraping → Grafana Visualization.