Resolved

Python asyncio watcher floods console with repetitive logging noise

Logging Optimization
Python
asyncio
Debugging
Console Output
DevOps
Performance

File watcher system generated excessive INFO-level logs obscuring critical events | Implemented logging noise reduction by downgrading verbose logs to DEBUG and introducing aggregated summaries

In a Python-based video upload automation system using asyncio and watchdog, the file watcher component was generating excessive INFO-level logs, overwhelming console output and obscuring critical events such as upload errors and progress. This resulted in a flood of repetitive log messages (700+ lines), making it impossible to track real-time upload status and diagnose issues.

Issue

The console output was cluttered with repetitive log messages, including:

  • Per-file operations: Metadata loading, series detection, and thumbnail discovery were logged at INFO level for every video file.
  • Obscured critical events: Upload failures, quota warnings, and progress updates were buried under excessive noise.
  • Reduced usability: Developers could not easily identify errors or monitor system status in real-time.

Example log noise (repeated for every video file):

  • Found existing video: uploads/youtube/...
  • Loaded metadata from uploads/youtube/...
  • Detected series: ...

This cluttered output made it difficult to focus on meaningful events such as upload errors and quota status.

Culprit

The issue stemmed from logging at the INFO level for operational details that were better suited for DEBUG-level logging. Three primary components contributed to the noise:

  • Metadata loading: Multiple INFO logs for each file’s metadata and series detection.
  • File discovery: INFO logs for every detected video file.
  • Pair processing: INFO logs for pair creation and completion status.

The logging architecture lacked:

  • Log level differentiation: INFO-level logs for operational details overwhelmed critical events.
  • Aggregation for bulk operations: Per-file logs did not provide consolidated summaries.
  • File-based logging: Verbose details were not redirected to files, leaving the console cluttered.

Solution

1. Log Level Adjustments

Downgraded per-file operational logs from INFO to DEBUG to reduce console clutter while preserving critical events at INFO level.

2. File-Based Logging System

Implemented a file-based logging system to store detailed operational logs in daily markdown files, ensuring:

  • Console clarity: Only critical events remain visible.
  • Preserved details: Detailed logs are stored in structured files for later review.
  • Separation of concerns: Console output focuses on high-level summaries.

3. Aggregated Console Summaries

Replaced repetitive per-file logs with batch summaries that provide consolidated information, such as:

  • Number of videos scanned and queued.
  • Missing pairs and their statuses.
  • Visual dashboards for startup summaries.

4. Selective Console Output

Redirected verbose operational logs to file-only logging while keeping critical events (e.g., upload status, quota warnings) visible in the console.

Impact

  • Reduced console clutter: 700+ lines of noise reduced to ~15 lines (98% reduction).
  • Improved visibility: Critical events like upload errors and quota warnings are now easily identifiable.
  • Enhanced usability: Developers can focus on meaningful system status without sifting through excessive logs.
  • Performance improvement: Reduced I/O overhead by minimizing console logging.

The solution ensures that logging remains informative while maintaining console usability and system performance.